# Errors: Batch transform on fine-tuned models

**URL:** <https://discuss.huggingface.co/t/errors-batch-transform-on-fine-tuned-models/16057>\
**Category:** Amazon SageMaker\
**Created:** [March 25, 2022, 7:40am UTC](https://discuss.huggingface.co/t/errors-batch-transform-on-fine-tuned-models/16057 "2022-03-25T07:40:27Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sud](https://avatars.discourse-cdn.com/v4/letter/s/ecd19e/32.png) [@sud](https://discuss.huggingface.co/u/sud)\
**Post date:** [March 25, 2022, 7:40am UTC](https://discuss.huggingface.co/t/errors-batch-transform-on-fine-tuned-models/16057/1 "2022-03-25T07:40:28Z")

</div>

A. Batch transform on 1M rows

```auto
2022-03-25 06:34:31,078 [WARN] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Token indices sequence length is longer than the specified maximum sequence length for this model (528 > 512). Running this sequence through the model will result in indexing errors
2022-03-25 06:34:31,092 [WARN] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - /codebuild/output/src257227288/src/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [249,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
2022-03-25 06:34:31,106 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Prediction error
2022-03-25 06:34:31,107 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - Traceback (most recent call last):
2022-03-25 06:34:31,107 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 222, in handle
2022-03-25 06:34:31,107 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - response = self.transform_fn(self.model, input_data, content_type, accept)
2022-03-25 06:34:31,107 [INFO] W-model-1-stdout com.amazonaws.ml.mms.wlm.WorkerLifeCycle - File "/opt/conda/lib/python3.6/site-packages/sagemaker_huggingface_inference_toolkit/handler_service.py", line 181, in transform_fn

```

B. Batch transform on smaller dataset (10K rows)

```auto
2022-03-25T07:17:24.961:[sagemaker logs]: sagemaker-us-east-2-460XXXXXXX64/batch_transform/input_head/oot_head_data.jsonl: ClientError: 400
2022-03-25T07:17:24.961:[sagemaker logs]: sagemaker-us-east-2-460XXXXXXX64/batch_transform/input_head/oot_head_data.jsonl: 
2022-03-25T07:17:24.961:[sagemaker logs]: sagemaker-us-east-2-460XXXXXXX64/batch_transform/input_head/oot_head_data.jsonl: Message:
2022-03-25T07:17:24.961:[sagemaker logs]: sagemaker-us-east-2-460XXXXXXX64/batch_transform/input_head/oot_head_data.jsonl: {
2022-03-25T07:17:24.961:[sagemaker logs]: sagemaker-us-east-2-460XXXXXXX64/batch_transform/input_head/oot_head_data.jsonl: "code": 400,
2022-03-25T07:17:24.961:[sagemaker logs]: sagemaker-us-east-2-460XXXXXXX64/batch_transform/input_head/oot_head_data.jsonl: "type": "InternalServerException",
2022-03-25T07:17:24.961:[sagemaker logs]: sagemaker-us-east-2-460XXXXXXX64/batch_transform/input_head/oot_head_data.jsonl: "message": "CUDA error: device-side assert triggered"

2022-03-25 07:17:24,943 [WARN] W-model-1-stderr com.amazonaws.ml.mms.wlm.WorkerLifeCycle - /codebuild/output/src257227288/src/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [251,0,0], thread: [35,0,0] Assertion `srcIndex < srcSelectDimSize` failed.

```

The code that I am using to create batch transform is as follows

```auto
sagemaker_session_bucket = sess.default_bucket()
df_processed_oot['inputs'] = df_processed_oot['detailedissue']
df_processed_oot[['inputs']].head(10000).to_csv(config['oot_head_csv'], index=None)
# datset files
dataset_csv_file = config['oot_head_csv']
dataset_jsonl_file = "oot_head_data.jsonl"

with open(dataset_csv_file, "r+") as infile, open(dataset_jsonl_file, "w+") as outfile:
    reader = csv.DictReader(infile);
    for row in reader:
        # remove @
        #row["inputs"] = row["inputs"].replace("@","")
        json.dump(row, outfile);
        outfile.write('\n');

input_s3_path = s3_path_join("s3://", sagemaker_session_bucket, "batch_transform/input_head")
output_s3_path = s3_path_join("s3://", sagemaker_session_bucket, "batch_transform/output_head")
s3_file_uri_head = S3Uploader.upload(dataset_jsonl_file, input_s3_path)

print(f"{dataset_jsonl_file} uploaded to {s3_file_uri}")

# create Hugging Face Model Class for classifier
huggingface_model = HuggingFaceModel(
   model_data =model_uri, # configuration for loading model from Hub
   role=role, # iam role with permissions to create an Endpoint
   transformers_version="4.6", # transformers version used
   pytorch_version="1.7", # pytorch version used
   py_version='py36', # python version used
)

# create Transformer to run our batch job
batch_job = huggingface_model.transformer(
    instance_count=1,
    instance_type='ml.g4dn.xlarge',
    output_path=output_s3_path, # we are using the same s3 path to save the output with the input
    strategy='SingleRecord'
)

batch_job.transform(data=s3_file_uri, content_type='application/json', split_type='Line')

```

I am new to HF and transformer libraries; it would be great if someone could help me out here with the easiest way to use the truncate option during batch prediction on a large dataset

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 25, 2022, 8:17am UTC](https://discuss.huggingface.co/t/errors-batch-transform-on-fine-tuned-models/16057/2 "2022-03-25T08:17:59Z")

</div>

> [@sud](#):
>
> ```auto
> Token indices sequence length is longer than the specified maximum sequence length for this model (528 > 512). Running this sequence through the model will result in indexing errors
> 
> ```

This error says that you are providing inputs that are longer than the model can handle. You can control this either through additional `parameters` in the `jsonl` file, but for this to be possible the model/tokenizer must be able to know the max\_length of the model.

---

<div class="post-metadata">

**Author:** ![sud](https://avatars.discourse-cdn.com/v4/letter/s/ecd19e/32.png) [@sud](https://discuss.huggingface.co/u/sud)\
**Post date:** [March 25, 2022, 10:28am UTC](https://discuss.huggingface.co/t/errors-batch-transform-on-fine-tuned-models/16057/3 "2022-03-25T10:28:47Z")

</div>

> [@sud](#):
>
> `Token indices sequence length is longer than the specified maximum sequence length for this model (528 > 512). Running this sequence through the model will result in indexing errors`

Thanks for the prompt reply. Is there a sample notebook that helps add parameters when we convert csv into a jsonl file for each record?

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [March 25, 2022, 12:29pm UTC](https://discuss.huggingface.co/t/errors-batch-transform-on-fine-tuned-models/16057/4 "2022-03-25T12:29:14Z")

</div>

You can take a look here: [Reference](https://huggingface.co/docs/sagemaker/reference#inference-toolkit-api)

---

<div class="post-metadata">

**Author:** ![jenpeper](https://avatars.discourse-cdn.com/v4/letter/j/8e7dd6/32.png) [@jenpeper](https://discuss.huggingface.co/u/jenpeper)\
**Post date:** [May 4, 2023, 1:33pm UTC](https://discuss.huggingface.co/t/errors-batch-transform-on-fine-tuned-models/16057/5 "2023-05-04T13:33:32Z")

</div>

I ran into a similar issue, the data in your jsonl file for inference should look something like this…

```auto
{"inputs":"...Some long text string, likely over 512 tokens after tokenization....","parameters":{"truncation":true,"max_length":512}}
{"inputs":"...Another long text string, likely over 512 tokens after tokenization....","parameters":{"truncation":true,"max_length":512}}

```