# Longformer and sentiment analysis

**URL:** <https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416>\
**Category:** Beginners\
**Created:** [August 23, 2021, 5:51pm UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416 "2021-08-23T17:51:44Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![codedancer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/codedancer/32/2549_2.png) [@codedancer](https://discuss.huggingface.co/u/codedancer)\
**Post date:** [August 23, 2021, 5:51pm UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416/1 "2021-08-23T17:51:44Z")

</div>

I am trying to use longformer to do a sentiment analysis and I am wondering what the best way is to do it. I have the following code:

> from transformers import LongformerTokenizer, EncoderDecoderModel  
> model = EncoderDecoderModel.from\_pretrained(“patrickvonplaten/longformer2roberta-cnn\_dailymail-fp16”)  
> tokenizer = LongformerTokenizer.from\_pretrained(“allenai/longformer-base-4096”)

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [August 23, 2021, 6:58pm UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416/2 "2021-08-23T18:58:46Z")

</div>

Hi,

LongFormer itself is a Transformer encoder, and that’s more than sufficient to perform sentiment analysis. You can just use `LongFormerForSequenceClassification`, like so:

```auto
from transformers import LongformerTokenizer, LongformerForSequenceClassification
import torch

tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')
model = LongformerForSequenceClassification.from_pretrained('allenai/longformer-base-4096')

inputs = tokenizer("This text is positive", return_tensors="pt")
labels = torch.tensor([1]).unsqueeze(0) # Batch size 1
outputs = model(**inputs, labels=labels)
loss = outputs.loss
logits = outputs.logits

```

Note that this model will have a classification head that is randomly initialized, so you’ll need to fine-tune it on a custom dataset.

---

<div class="post-metadata">

**Author:** ![codedancer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/codedancer/32/2549_2.png) [@codedancer](https://discuss.huggingface.co/u/codedancer)\
**Post date:** [August 23, 2021, 7:08pm UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416/3 "2021-08-23T19:08:05Z")

</div>

This is great! I think I am getting it but I am just wondering how you get the label and socre then. I normally use pipeline and sentiment-analysis and I can get something like [{‘label’: ‘NEGATIVE’, ‘score’: 0.9982099533081055}].

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [August 24, 2021, 7:00am UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416/4 "2021-08-24T07:00:06Z")

</div>

So you want to do inference with an already fine-tuned model?

Looking on the [hub](https://huggingface.co/models?other=longformer&pipeline_tag=text-classification&sort=downloads), there are currently no LongFormer checkpoints fine-tuned on a sentiment analysis dataset. So feel free to upload the first LongFormer checkpoint fine-tuned on a sentiment analysis dataset to the hub 😉

Once you’ve trained a model, you can plug it into the pipeline API for quick inference.

Here’s an example with a RoBERTa model from the hub, fine-tuned on sentiment analysis:

```auto
from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline

model_name = "cardiffnlp/twitter-roberta-base-sentiment"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

nlp = pipeline('sentiment-analysis', model=model, tokenizer=tokenizer)

text = "this is a very positive text"

outputs = nlp(text)

```

In this case, printing the outputs returns `[{'label': 'LABEL_2', 'score': 0.9623518586158752}]`. This is of course not that helpful, as we don’t know what LABEL\_2 means. This is because the authors of this model have not updated the `config.id2label`. However, looking at [this file](https://raw.githubusercontent.com/cardiffnlp/tweeteval/main/datasets/sentiment/mapping.txt) (mentioned in the code example on the [model page](https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment)), it appears that LABEL\_2 means “positive”.

---

<div class="post-metadata">

**Author:** ![codedancer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/codedancer/32/2549_2.png) [@codedancer](https://discuss.huggingface.co/u/codedancer)\
**Post date:** [August 24, 2021, 11:32am UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416/5 "2021-08-24T11:32:23Z")

</div>

Thanks @nielsr! I think I’d need to train one and if I get to finish that, I’ll share it out. I guess the other way to tackle the longform text is to focus on the maximum limit of the text the pre-trained model can take. Do you happen to know what the limit is - like how much text I can push into the pipeline for the sentiment?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [August 24, 2021, 7:00pm UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416/6 "2021-08-24T19:00:48Z")

</div>

Models like BERT, RoBERTa, etc. all take a max sequence length of 512 tokens. Note that these models use _subword_ tokenization, which means that a given word might be tokenized into several tokens, so in practice these models can take in less than 500 words. So if you really want to use the pipeline API with a very long text, you can use models like [LongFormer](https://huggingface.co/transformers/model_doc/longformer.html) or [BigBird](https://huggingface.co/transformers/model_doc/bigbird.html), which can handle 4096 tokens in a single forward pass.

As an alternative (if you still want to use a model like BERT/RoBERTa), you can implement it using a sliding window approach. You can do this using the tokenizer, like so:

```auto
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

text = "a very long text"
encoding = tokenizer(text, return_overflowing_tokens=True, stride=128)

```

This will create multiple training examples (encodings) for a given text, by sliding a window (each having 128 tokens of overlap) across the text. You can then feed each training example through the model, and average the predictions.

---

<div class="post-metadata">

**Author:** ![shensmobile](https://avatars.discourse-cdn.com/v4/letter/s/7ea924/32.png) [@shensmobile](https://discuss.huggingface.co/u/shensmobile)\
**Post date:** [August 30, 2023, 11:49pm UTC](https://discuss.huggingface.co/t/longformer-and-sentiment-analysis/9416/7 "2023-08-30T23:49:48Z")

</div>

Hi nielsr,

Sorry to bother you on this! If I’m looking to utilize RoBERTa for long sequence sentiment analysis and I want to use tokenizer stride during training, do I need to write an entirely custom train() logic to average the predictions during the batch evaluation? Or is there something already built-in to trainer.train() that I could leverage to take advantage of strides?