# ClientErro:400 when using batch transformer for inference **URL:** \ **Category:** Amazon SageMaker\ **Created:** [January 7, 2022, 10:06pm UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476 "2022-01-07T22:06:43Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![miOmiO](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@miOmiO](https://discuss.huggingface.co/u/miOmiO)\ **Post date:** [January 7, 2022, 10:06pm UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/1 "2022-01-07T22:06:43Z")
Hi everyone, I try to do sentiment analysis on a bunch of data and follow the example notebook [notebooks/sagemaker-notebook.ipynb at main · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/master/sagemaker/12_batch_transform_inference/sagemaker-notebook.ipynb) and below is my code: ```auto from sagemaker.huggingface.model import HuggingFaceModel hub = { 'HF_MODEL_ID':'cardiffnlp/twitter-roberta-base-sentiment', 'HF_TASK':'text-classification' } # create Hugging Face Model Class huggingface_model = HuggingFaceModel( env=hub, role=role, transformers_version='4.6', pytorch_version="1.7", py_version='py36', ) # create Transformer to run our batch job batch_job = huggingface_model.transformer( instance_count=1, instance_type='ml.p3.2xlarge', output_path=output_s3_path, strategy='SingleRecord') # starts batch transform job and uses s3 data as input batch_job.transform( data=input_s3_path, content_type="application/json", split_type="Line") ``` My input jsonl file is formatted as ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/8/87868e580ac2423368623c30d89992de6973832c.png) The model successfully infer when I feed with first 10 rows (totally this data is about 6k rows) but it throw errors when I expand to 100 rows. I’m super new to sagemaker and hugging face , can anyone tell me what I’m missing ? Thank you ---
**Author:** ![miOmiO](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@miOmiO](https://discuss.huggingface.co/u/miOmiO)\ **Post date:** [January 7, 2022, 10:11pm UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/2 "2022-01-07T22:11:32Z")
sorry it’s still me, just attach more pics so the context might be better understood. I heard BERT has limitation on text length so I truncated each line to 460 words ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/706ba3e693996eebc67e92dfef1f4cf86f32b956.png) ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [January 10, 2022, 11:14am UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/3 "2022-01-10T11:14:49Z")
Hi miOmiO - did the truncation fix your problem? If not, would you mind sharing what error message you are getting? ---
**Author:** ![miOmiO](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@miOmiO](https://discuss.huggingface.co/u/miOmiO)\ **Post date:** [January 10, 2022, 5:10pm UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/4 "2022-01-10T17:10:01Z")
Hi @marshmellow77 , yes, truncation make some long text acceptable to the model, but not all of them, even I specify the length to 460 words. I tried from 500 to 460. I have no idea if I should keep reducing this or other modification would help. ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/3/3f9040fced1539ea346d1961d219f3c9c1da415c.png) ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [January 10, 2022, 5:27pm UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/5 "2022-01-10T17:27:38Z")
@miOmiO you have created a second thread: [ClientError:400 when using batch transformer on sagemaker for inference](https://discuss.huggingface.co/t/clienterror-400-when-using-batch-transformer-on-sagemaker-for-inference/13481) where i responded. Can this be closed? or those two different threads. ---
**Author:** ![miOmiO](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@miOmiO](https://discuss.huggingface.co/u/miOmiO)\ **Post date:** [January 10, 2022, 7:47pm UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/6 "2022-01-10T19:47:03Z")
Hi @philschmid , sure, please kindly close this post, I will update on that one. Thank you. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [January 11, 2022, 7:11am UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/7 "2022-01-11T07:11:09Z")
Thanks for letting me know. Then here again the response > Hey @miOmiO, > > Happy to help you here! to narrow down your issue. I think the first step would be to check if the dataset was created in the sample ([notebooks/sagemaker-notebook.ipynb at master · huggingface/notebooks · GitHub](https://github.com/huggingface/notebooks/blob/master/sagemaker/12_batch_transform_inference/sagemaker-notebook.ipynb)) works or if it also errors out. > > Additionally, could you bump the version of the `HuggingFaceModel` to the latest one? For `transformers_version` that’s `4.12.3` and for `pytorch_version` its `1.9.1` maybe this already solves your issue. You can find the list of available containers here: [Reference](https://huggingface.co/docs/sagemaker/reference#inference-dlc-overview) > > Also worth testing is to replace your model with a different model, e.g. `distilbert-base-uncased-finetuned-sst-2-english` ---
**Author:** ![miOmiO](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@miOmiO](https://discuss.huggingface.co/u/miOmiO)\ **Post date:** [January 11, 2022, 7:03pm UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/8 "2022-01-11T19:03:19Z")
Hi @philschmid , Thank you for the reply! I check all three approaches : 1 The inputs format comply with the json format from sample notebook. 2 I use the latest version of of framework as suggested 3 Also switch to different model to see if it’s specific model issue. It still prompts client error:400 , but I notice that as long as I truncated my each line of text to 60 words(randomly pick the number) , both two model work fine. Is this related to the text length limitation? If so, is there any way I can specify the length of text when building the batch job? P.S. my data set is around 6k rows , some rows have more than 512 words/tokens. ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [January 11, 2022, 8:31pm UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/9 "2022-01-11T20:31:51Z")
Hi @miOmiO - text length limitation could be an issue here. Note that the length refers to the number of tokens, not the number of words. Because BERT models generally use subword tokenization it can happen that one word is split into 2 or more tokens. That is why even reducing the number of words to 460 sometimes might throw an error. To test this you could try to use the model row by row and see if the number of examples that fail correspond to the same ones in your batch job. If it is indeed the number of tokens that cause the model to fail you should be seeing an error message like` "... sequence length is longer than the specified maximum sequence length for this model ..."` If this indeed the source of error then it might be easiest to truncate the input sequence of tokens after the tokenization (rather than the number of words before tokenization). Hope that helps. ---
**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\ **Post date:** [January 12, 2022, 7:15am UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/10 "2022-01-12T07:15:22Z")
The model `cardiffnlp/twitter-roberta-base-sentiment` don’t has a `max len` defined. I tried to reach out the authors but they haven’t responded. See [Add `tokenizer_max_length` to `cardiffnlp/twitter-roberta-base-sentiment` · Issue #13459 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/13459) You could “fork” the model → creating a new model repository and push the weights + add the `tokenizer_config` then `truncation:True` should work properly. ---
**Author:** ![miOmiO](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@miOmiO](https://discuss.huggingface.co/u/miOmiO)\ **Post date:** [January 13, 2022, 4:47am UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/11 "2022-01-13T04:47:22Z")
Hi @marshmellow77 , thank you for leading me into thought of sub-word tokenization . ---
**Author:** ![miOmiO](https://avatars.discourse-cdn.com/v4/letter/m/53a042/32.png) [@miOmiO](https://discuss.huggingface.co/u/miOmiO)\ **Post date:** [January 13, 2022, 4:52am UTC](https://discuss.huggingface.co/t/clienterro-400-when-using-batch-transformer-for-inference/13476/12 "2022-01-13T04:52:06Z")
Hi @philschmid , I got solution from another related post under your help as well : [How are the inputs tokenized when model deployment? - Amazon SageMaker - Hugging Face Forums](https://discuss.huggingface.co/t/how-are-the-inputs-tokenized-when-model-deployment/9692/12) After I switch to another model and remodify the input json file format like this: {“inputs”: “long sentence 2”, “parameters”: {“trucation”: true}}, the new model works well for me ( as long as it has ‘max\_length’ attribute in tokenizer config file.)