# Text Length FinBert - Serverless Inference Endpoint

**URL:** <https://discuss.huggingface.co/t/text-length-finbert-serverless-inference-endpoint/18496>\
**Category:** Amazon SageMaker\
**Created:** [May 29, 2022, 9:43pm UTC](https://discuss.huggingface.co/t/text-length-finbert-serverless-inference-endpoint/18496 "2022-05-29T21:43:29Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![thanksfinance](https://avatars.discourse-cdn.com/v4/letter/t/ea666f/32.png) [@thanksfinance](https://discuss.huggingface.co/u/thanksfinance)\
**Post date:** [May 29, 2022, 9:43pm UTC](https://discuss.huggingface.co/t/text-length-finbert-serverless-inference-endpoint/18496/1 "2022-05-29T21:43:29Z")

</div>

Hi guys, I’m trying to send a long text (longer than 512) to a FinBert inference endpoint deployed on a serveless inference endpoint on aws.  
I’m receiving the following error: “The size of tensor a (639) must match the size of tensor b (512) at non-singleton dimension 1”.

I have a list of text that I would like to classify without splitting them, how can I fix?

Thank you in advance

---

<div class="post-metadata">

**Author:** ![philschmid](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/philschmid/32/38022_2.png) [@philschmid](https://discuss.huggingface.co/u/philschmid)\
**Post date:** [May 30, 2022, 6:32am UTC](https://discuss.huggingface.co/t/text-length-finbert-serverless-inference-endpoint/18496/2 "2022-05-30T06:32:14Z")

</div>

The model has a `max_sequence_length` of `512`. You can provide `truncation=True` as parameter, e.g.

```auto
{
  "inputs": "Hi, I recently bought a device from your company but it is not working as advertised and I would like to get reimbursed!",
  "parameters": {
   "truncation": True
  }
}

```

---

<div class="post-metadata">

**Author:** ![thanksfinance](https://avatars.discourse-cdn.com/v4/letter/t/ea666f/32.png) [@thanksfinance](https://discuss.huggingface.co/u/thanksfinance)\
**Post date:** [November 4, 2022, 11:30pm UTC](https://discuss.huggingface.co/t/text-length-finbert-serverless-inference-endpoint/18496/3 "2022-11-04T23:30:58Z")

</div>

Hi @philschmid  
I am testing several pre-trained models that I find on the hub for Text-Classification. Many have a max\_length=512. I deploy them to SageMaker Endpoint Serverless and invoke them from a lambda.

Months ago you suggested me to use the truncation parameter… Now… I was wondering…  
if the text is longer, is it truncated? Do I then lose the information about the “excess” part or is each batch sentence evaluated and a result for the whole document processed?

Is there a way to define a preprocessing operation to chunk the sentence in order to get a better evaluation?

Something like:

```auto
tokenizer = BertTokenizer.from_pretrained(MODEL_ID)
tokens = tokenizer.encode_plus(text, add_special_tokens = False, return_tensors='pt')
input_id_chunks = tokens['input_ids'][0].split(510)
mask_chunks = tokens['attention_mask'][0].split(510)
chunksize = 512

input_id_chunks = list(input_id_chunks)
mask_chunks = list(mask_chunks)

for i in range(len(input_id_chunks)):
    input_id_chunks[i] = torch.cat([
        torch.Tensor([101]), input_id_chunks[i].float(), torch.Tensor([102])
    ])
    mask_chunks[i] = torch.cat([
        torch.Tensor([1]), mask_chunks[i].float(), torch.Tensor([1])
    ])
    
    pad_len = chunksize - input_id_chunks[i].shape[0]
    if pad_len > 0 :
        input_id_chunks[i] = torch.cat([
            input_id_chunks[i], torch.Tensor([0] * pad_len)
        ])
        mask_chunks[i] = torch.cat([
            mask_chunks[i], torch.Tensor([0] * pad_len)
        ])

```

Can I also know where I find the parameters that I can pass to my inference endpoint as input? Is there a resource that you can link?

---

<div class="post-metadata">

**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\
**Post date:** [November 5, 2022, 7:22am UTC](https://discuss.huggingface.co/t/text-length-finbert-serverless-inference-endpoint/18496/4 "2022-11-05T07:22:07Z")

</div>

Hi @thanksfinance. Yes, if you use the truncation parameter the text will be truncated and you will lose the “excess” part.

However, in text classification this is rarely a problem because the model is often able to determine the class with just using the first 512 tokens. Do you see a significant deterioration in your metrics when using the truncation parameter?

If yes, you might indeed want to do some preprocessing. I’m not entirely clear what your code does, but this [thread](https://stackoverflow.com/questions/58636587/how-to-use-bert-for-long-text-classification) discusses a similar issue and potential options how to solve this challenge.