# T5: classification using text2text?

**URL:** <https://discuss.huggingface.co/t/t5-classification-using-text2text/504>\
**Category:** 🤗Transformers\
**Created:** [July 28, 2020, 6:54pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504 "2020-07-28T18:54:25Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [July 28, 2020, 6:54pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/1 "2020-07-28T18:54:25Z")

</div>

My friend who wishes to remain anonymous asked a good question about T5 that I couldn’t answer:

Say we have a model that predicts sentiment – answers are “positive/negative/neutral” – for something like RoBERTa we’d add a layer, slap on a softmax – and we get both argmax predictions, and some notion of probability amongst the three classes (as well as entropy).

For T5, we just get a text reply.

But of course if we looked at the outputs to the softmax in T5, we’d see p(“positive”) etc – assuming the response is 1 token.

Has anyone tried to do this already or seen examples/notebooks like this?

Ideally, we want to ask our model several questions. Without worrying too much about the conditional logic, we’d like to be able to measure the probability of text outputs, including some rare categories (that nonetheless are present in our training set). As well as to look for low and high entropy predictions.

If nobody has done this, any code pointers where to look would be helpful.

---

<div class="post-metadata">

**Author:** ![thomwolf](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thomwolf/32/2198_2.png) [@thomwolf](https://discuss.huggingface.co/u/thomwolf)\
**Post date:** [July 29, 2020, 9:07am UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/2 "2020-07-29T09:07:29Z")

</div>

I’m not sure I understand what is the difference between what you are describing and how the GLUE dataset is handled in the T5 paper

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [July 29, 2020, 12:33pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/3 "2020-07-29T12:33:02Z")

</div>

I’m also not sure what the question means here, are you trying to ask if someone has used T5 for classification ?Then yes, I’ve fine-tuned it for both binary and multi-class classification [here](https://github.com/patil-suraj/exploring-T5/blob/master/t5_fine_tuning.ipynb).

As for measuring the probability, this [paper](https://arxiv.org/pdf/2003.06713.pdf) used T5 in really interesting way for document ranking. First they train the model to predict `true` if doc is related to the query and `false` if not. And for ranking they apply a softmax only on the logits of the “true” and “false” token and rerank using the probabilities assigned to the “true” token.

.

---

<div class="post-metadata">

**Author:** ![ashispapu](https://avatars.discourse-cdn.com/v4/letter/a/eb8c5e/32.png) [@ashispapu](https://discuss.huggingface.co/u/ashispapu)\
**Post date:** [August 2, 2020, 5:55am UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/4 "2020-08-02T05:55:18Z")

</div>

To answer first part of your question, Yes, I have tried T5 for multi class classification. It generates the tokens based on the class types(It could be a single token or multiple tokens based on the tokenization of the class label)

Second part of the question is not clear to me. Please explain more.

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 24, 2020, 8:25pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/5 "2020-09-24T20:25:31Z")

</div>

Got a more specific version of the question:

> How should I fix this code to use t5 without finetuning for sentiment:

```auto
from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")

input_sent = [' sentiment review: I love chocolate', ' sentiment review: I love chocolate']
labels = ['positive ', 'negative ']
input_ids = tokenizer(input_sent, return_tensors='pt').to(torch_device).input_ids
target_ids = tokenizer(labels, return_tensors='pt').to(torch_device).input_ids
#target_ids = torch.tensor([target_ids])
outputs = model(input_ids=input_ids, labels=target_ids, use_cache=False, return_dict=True)

```

I got the sentiment review prefix from [T5 to sentiment analysis · Issue #109 · google-research/text-to-text-transfer-transformer · GitHub](https://github.com/google-research/text-to-text-transfer-transformer/issues/109) @valhalla

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 25, 2020, 6:32am UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/6 "2020-09-25T06:32:55Z")

</div>

T5 was trained on `sst2` as part of it’s multi-task pre-training mixture, so to use T5 for sentiment without fine-tuning use the prefix `sst2 sentence: ` and pass it to the model. You can do it two ways

```auto
from transformers import T5ForConditionalGeneration, T5Tokenizer
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")

text = "sst2 sentence: it confirms fincher ’s status as a film maker who artfully bends technical know-how to the service of psychological insight"

enc = tokenizer(text, return_tensors="pt")
tokens = model.generate(**enc)
tokenizer.batch_decode(tokens)
=> ['positive']

```

or use the `text2text-generation` pipeline

```auto
t5_sentiment = pipeline("text2text-generation")
text = "sst2 sentence: it confirms fincher ’s status as a film maker who artfully bends technical know-how to the service of psychological insight"
t5_sentiment(text)
=> [{'generated_text': 'positive'}]
```

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 25, 2020, 2:38pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/7 "2020-09-25T14:38:27Z")

</div>

That’s cool suraj, thanks! Is it possible to do that with forward?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [September 25, 2020, 2:52pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/8 "2020-09-25T14:52:31Z")

</div>

forward as in without using `generate`?

As T5 is trained using text-2-text approach we need to generate the output as text either manually calling `forward` or using `generate`. If we wish to do this as discriminative task we could take the same approach as `BART` where we feed the same text to both `encoder` and `decoder`, pool the hidden states of the final `eos` token and pass that to a classification head, this is how `BartForSequecneClassfication` works. Not sure how this will work for T5, haven’t tried myself.

To answer the original question, you could use forward as shown below to generate the output

```python
import torch
import torch.nn.functional as F
from transformers import T5ForConditionalGeneration, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")
model.eval()

text = "sst2 sentence: it confirms fincher ’s status as a film maker who artfully bends technical know-how to the service of psychological insight"
with torch.no_grad():
  enc = tokenizer(text, return_tensors="pt")
  decoder_input_ids = torch.tensor([tokenizer.pad_token_id]).unsqueeze(0) 
  logits = model(**enc, decoder_input_ids=decoder_input_ids)[0]
  tokens = torch.argmax(logits, dim=2)
  sentiments = tokenizer.batch_decode(tokens)
  # 'positve'

```

Now if we wish to measure the probabilities, as I described in the earlier comment, we could only take the `logits` of `positive` and `negative` token and apply `softmax` on it. Thankfully T5 encodes `positive` and `negative` as single tokens so it’s easy to do. The token id for positive is 1465 and for negative 2841.

```python
logits = logits.squeeze(1)
# only take the logits of positive and negative
selected_logits = logits[:, [1465, 2841]] 

probs = F.softmax(selected_logits, dim=1)
#=> tensor([[0.9820, 0.0180]])

```

Hope this answers your question.

cc @sshleifer

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [September 25, 2020, 6:25pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/9 "2020-09-25T18:25:25Z")

</div>

Incredible answer, thanks a ton!

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [June 18, 2021, 10:32am UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/10 "2021-06-18T10:32:49Z")

</div>

Sorry for the topic steal, wasn’t getting a lot of attention on my topic on T5 ( [Yet another question about T5 prefixes: are they special? - Models - Hugging Face Forums](https://discuss.huggingface.co/t/yet-another-question-about-t5-prefixes-are-they-special/6438)).

Has anyone here used T5 for regression? From the paper that @valhalla links, it seems that you could rebase your continuous labels to 0-1 and then use the output of the softmax for one of two options (e.g. true and false) in a MSELoss function. Or is that a nonsense suggestion? The caveat is that it is not always possible to rebase your values to 0-1.

---

<div class="post-metadata">

**Author:** ![dipanjann](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dipanjann/32/3016_2.png) [@dipanjann](https://discuss.huggingface.co/u/dipanjann)\
**Post date:** [November 3, 2021, 9:34am UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/11 "2021-11-03T09:34:02Z")

</div>

Thanks ! This helps a great Deal . I wanted to know how to make use of this if the token\_id of the word is segregated into 2 or more parts . like for entailment , the token\_id in t5 model is [35,5756,297] .

How can i fit this in selected\_logits = logits[:, [1465, 2841]] .  
I used using array of array’s . Am I missing any tweak?

---

<div class="post-metadata">

**Author:** ![santon](https://avatars.discourse-cdn.com/v4/letter/s/3da27b/32.png) [@santon](https://discuss.huggingface.co/u/santon)\
**Post date:** [November 11, 2022, 7:11pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/12 "2022-11-11T19:11:56Z")

</div>

@dipanjann I’ve wondered the same thing. My solution when the output text corresponds to multiple tokens is the following. I’m not sure it’s completely correct, but the basic idea is to use `generate()` to get the prediction logits for each step in the generated output sequence. Next, you roughly calculate the cross entropy loss for each of the possible output classes.

I’m really not sure if this is exactly mathematically correct. I’d love if someone can show a better way! In my tests, it produces reasonable results.

```python
import typing as T

import pytorch_lightning
import torch
from more_itertools import chunked

class MyLitModule(pytorch_lightning.LightningModule):
    ...

    def predict_proba(self, text: T.Iterable[str], labels: T.Iterable[str]):
        """Predict the class probabilities"""
        # Compute the tokens corresponding to the text labels:
        class_ids = torch.LongTensor(self.tokenizer(list(labels), padding=True).input_ids)

        logits = []
        for chunk in chunked(text, 16):
            # Tokenize the input text:
            encoding = self.tokenizer(
                list(chunk),
                max_length=self.hparams.model_max_length,
                padding=True,
                truncation=True,
                return_tensors="pt",
            )

            output_sequences = self.model.generate(
                input_ids=encoding.input_ids.to(self.device),
                attention_mask=encoding.attention_mask.to(self.device),
                do_sample=False,
                return_dict_in_generate=True,
                output_scores=True,
                min_length=class_ids.shape[1] + 1, max_length=class_ids.shape[1] + 1
            )

            # Generate the logits for each token in the generated output sequence.
            # `scores` has size [batch, seq_length, vocab_size]
            scores = torch.stack(output_sequences.scores, dim=1).to("cpu")
            # We don't care about the logits of special tokens:
            scores[:, :, self.tokenizer.all_special_ids] = torch.nan
            # Index the logits in `scores` based on the class token IDs.
            # For example, if class_ids[0, :] is [10, 30], then the prediction logits
            # are scores[:, 0, 10] and scores[:, 1, 30].
            # Finally, we average the logits, which is similar to how the cross entropy loss is calculated.
            logits.append(scores.gather(dim=2, index=class_ids.T.expand(len(chunk), -1, -1)))

        return torch.concat(logits, dim=0).nanmean(dim=1).softmax(1)

```

---

<div class="post-metadata">

**Author:** ![SaraAmd](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@SaraAmd](https://discuss.huggingface.co/u/SaraAmd)\
**Post date:** [June 17, 2023, 3:33am UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/13 "2023-06-17T03:33:29Z")

</div>

Thank you so much for the answer. I am doing a multi-class classification and the class labels are divided into subtokens. Do you know how I can get the token ids for each class and how to average them and then use a sofmaxt? I sincerely appreciate if you help me with this task

---

<div class="post-metadata">

**Author:** ![SaraAmd](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@SaraAmd](https://discuss.huggingface.co/u/SaraAmd)\
**Post date:** [June 20, 2023, 3:14pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/14 "2023-06-20T15:14:31Z")

</div>

I am trying you r code but i get an error at line " logits.append(scores.gather(dim=2, index=class\_ids.T.expand(len(chunk), -1, -1))) can you tell me why you use gather? and my text is not divided into chunks so how can I GET THE PROBABILITIES DIRECTLY.Can you help me with this please

---

<div class="post-metadata">

**Author:** ![SaraAmd](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@SaraAmd](https://discuss.huggingface.co/u/SaraAmd)\
**Post date:** [June 20, 2023, 4:39pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/15 "2023-06-20T16:39:31Z")

</div>

> [@santon](#):
>
> `torch.concat(logits, dim=0).nanmean(dim=1).softmax(1)`

@santon My class labels have multiple tokens and it seems it does not work for me. I have three classes but the output tensor after the softmax is [6 3] which does not make sense. Can anyone help me with this task

---

<div class="post-metadata">

**Author:** ![SaraAmd](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@SaraAmd](https://discuss.huggingface.co/u/SaraAmd)\
**Post date:** [June 22, 2023, 8:09pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/16 "2023-06-22T20:09:23Z")

</div>

To me .expand does not make sense because you do not want the logits for the input seq, you just need it for the class labels

---

<div class="post-metadata">

**Author:** ![devvanshhh](https://avatars.discourse-cdn.com/v4/letter/d/c0e974/32.png) [@devvanshhh](https://discuss.huggingface.co/u/devvanshhh)\
**Post date:** [November 16, 2023, 11:18am UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/17 "2023-11-16T11:18:29Z")

</div>

Has anyone tried multiple classifications in the same query? In the context of this problem it might look something like,  
“This product looks good, I might buy it.” → [“positive”, “will buy”]  
“This product looks good, but too expensive.” → [“positive”, “won’t buy”]  
I want to try having various fields like these and want the model to classify all the fields in the same query. Is this possible/ feasible?

---

<div class="post-metadata">

**Author:** ![MattiLinnanvuori](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MattiLinnanvuori](https://discuss.huggingface.co/u/MattiLinnanvuori)\
**Post date:** [November 18, 2023, 10:20am UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/18 "2023-11-18T10:20:48Z")

</div>

Yes, it is possible and called multi-label classification. See e.g. the following link. [https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine\_tuning\_BERT\_(and\_friends)\_for\_multi\_label\_text\_classification.ipynb](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/BERT/Fine_tuning_BERT_(and_friends)_for_multi_label_text_classification.ipynb)

---

<div class="post-metadata">

**Author:** ![SaraAmd](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@SaraAmd](https://discuss.huggingface.co/u/SaraAmd)\
**Post date:** [January 23, 2024, 10:06pm UTC](https://discuss.huggingface.co/t/t5-classification-using-text2text/504/19 "2024-01-23T22:06:46Z")

</div>

@ dipanjann Did you figure it out? I have the same question. My class labels have sub tokens and I dont know ow to handle them