# InternalServer Exception when deploying fine tuned model on Sagemaker

**URL:** <https://discuss.huggingface.co/t/internalserver-exception-when-deploying-fine-tuned-model-on-sagemaker/9978>\
**Category:** Amazon SageMaker\
**Created:** [September 14, 2021, 12:23pm UTC](https://discuss.huggingface.co/t/internalserver-exception-when-deploying-fine-tuned-model-on-sagemaker/9978 "2021-09-14T12:23:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 14, 2021, 12:23pm UTC](https://discuss.huggingface.co/t/internalserver-exception-when-deploying-fine-tuned-model-on-sagemaker/9978/1 "2021-09-14T12:23:05Z")

</div>

Hello everyone,

I got a fine tuned german classification model which I want to deploy on Sagemaker. To test my model I want to predict a couple of texts in the json format. Here is my code:

```auto
import csv
import json
import sagemaker
from sagemaker.s3 import S3Uploader,s3_path_join

# get the s3 bucket
sess = sagemaker.Session()
role = sagemaker.get_execution_role()
sagemaker_session_bucket = sess.default_bucket()

# uploads a given file to S3.
input_s3_path = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/input")
output_s3_path = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/output")
#s3_file_uri = S3Uploader.upload(dataset_jsonl_file,input_s3_path)

# datset files
dataset_csv_file="test"
dataset_jsonl_file="test.jsonl"

with open(dataset_csv_file, "r+") as infile, open(dataset_jsonl_file, "w+") as outfile:
    reader = csv.DictReader(infile)
    for row in reader:
        # remove @
        json.dump(row, outfile)
        outfile.write('\n')

s3_file_uri = S3Uploader.upload(dataset_jsonl_file,input_s3_path)

#package my fine tuned model
!cd model_token && tar zcvf model.tar.gz * 
!mv model_token/model.tar.gz ./model.tar.gz

model_url = s3_path_join("s3://",sagemaker_session_bucket,"batch_transform/model")
print(f"Uploading Model to {model_url}")
model_uri = S3Uploader.upload('model.tar.gz',model_url)
print(f"Uploaded model to {model_uri}")

from sagemaker.huggingface.model import HuggingFaceModel

# create Hugging Face Model Class
huggingface_model = HuggingFaceModel(
   model_data=model_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord')

# starts batch transform job and uses s3 data as input
batch_job.transform(
    data=s3_file_uri,
    content_type='application/json',    
    split_type='Line')

```

When I execute that batch tranform job I get the following error:

“code”: 400,  
2021-09-14T12:01:09.338:[sagemaker logs]: sagemaker-eu-central-1-422630546972/batch\_transform/input/test.jsonl: “type”: “InternalServerException”,  
2021-09-14T12:01:09.338:[sagemaker logs]: sagemaker-eu-central-1-422630546972/batch\_transform/input/test.jsonl: “message”: “text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).”

What am I missing?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 14, 2021, 12:44pm UTC](https://discuss.huggingface.co/t/internalserver-exception-when-deploying-fine-tuned-model-on-sagemaker/9978/2 "2021-09-14T12:44:05Z")

</div>

Hey @marlon89,

could you share the cloudwatch logs? They should provide more information to why the batch transform job is failing.  
Additionally, can you share an example of your `test.jsonl`?

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 14, 2021, 1:11pm UTC](https://discuss.huggingface.co/t/internalserver-exception-when-deploying-fine-tuned-model-on-sagemaker/9978/3 "2021-09-14T13:11:21Z")

</div>

Sure! I am super new to the whole AWS environment but I try…

Here is a part of my test.json:

```auto
{"input": "Machen Sie so weiter...."}
{"input": "."}
{"input": "Bleiben Sie so wie bisher!"}
{"input": "Weiter machen"}
{"input": "Passt alles. "}
{"input": "F\u00fcr alle Vergleiche, die bei mir anstehen, benutze ich Ihre Plattform schon seit Jahren. Ein Beweis f\u00fcr meine Zufriedenheit. "}
{"input": "alles bestens, vielen Dank!"}

```

When I create a realtime endpoint with the predict function it works:

```auto
predictor = huggingface_model.deploy(
   initial_instance_count=1, 
   instance_type="ml.m5.xlarge"
)

data = {
   "inputs": "Weiter machen"
}
predictor.predict(data)

```

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [September 14, 2021, 1:22pm UTC](https://discuss.huggingface.co/t/internalserver-exception-when-deploying-fine-tuned-model-on-sagemaker/9978/4 "2021-09-14T13:22:32Z")

</div>

> [@marlon89](#):
>
> ```auto
> {"input": "Machen Sie so weiter...."}
> 
> ```

`input` needs to be `inputs`. Then it should work.

P.S. there might also be an additional issue with encoding, looking at `"F\u00fcr alle Vergleiche,` i guess your data is not properly encoded/decoded to `utf-8`

---

<div class="post-metadata">

**Author:** ![marlon89](https://avatars.discourse-cdn.com/v4/letter/m/6de8d8/32.png) [@marlon89](https://discuss.huggingface.co/u/marlon89)\
**Post date:** [September 14, 2021, 1:43pm UTC](https://discuss.huggingface.co/t/internalserver-exception-when-deploying-fine-tuned-model-on-sagemaker/9978/5 "2021-09-14T13:43:20Z")

</div>

It works! Thanks for your super fast help!!!