# LLM Zero shot-text classification - How do you answer multiple questions computationally efficiently?

**URL:** <https://discuss.huggingface.co/t/llm-zero-shot-text-classification-how-do-you-answer-multiple-questions-computationally-efficiently/65074>\
**Category:** 🤗Transformers\
**Created:** [December 8, 2023, 2:30pm UTC](https://discuss.huggingface.co/t/llm-zero-shot-text-classification-how-do-you-answer-multiple-questions-computationally-efficiently/65074 "2023-12-08T14:30:09Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![pj-ml](https://avatars.discourse-cdn.com/v4/letter/p/d6d6ee/32.png) [@pj-ml](https://discuss.huggingface.co/u/pj-ml)\
**Post date:** [December 8, 2023, 2:30pm UTC](https://discuss.huggingface.co/t/llm-zero-shot-text-classification-how-do-you-answer-multiple-questions-computationally-efficiently/65074/1 "2023-12-08T14:30:09Z")

</div>

I have been very impressed with some of the latest open-source 7B LLMs, like [OpenChat3.5](https://huggingface.co/openchat/openchat_3.5) and [OpenHermes2.5](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B).

I have been working on some code to classify text with these models by:

1. Asking questions about the input text with (specifying a set of expected responses or options)
2. Calculating the joint output probability of each expected response token sequence using the LLM output

The typical structure is something like:  
CLASSIFY\_PROMPT = “{instruction}{text\_to\_classify}{question\_and\_options}”

I ask a question with specific options as output. E.g. How did the reviewer find the movie? (good/bad/ok)

I then append each option (with the appropriate tokens to specify end of turn and end of sequence) to the CLASSIFY\_PROMPT and calculate the joint token probabilities of each output option sequence. My result is something like:  
good\<eos\_token\> - 92%  
ok\<eos\_token\> - 3%  
bad\<eos\_token\> - 0.001%

In the real-world scenario, my text\_to\_classify is more complex, and I want to ask multiple questions about it.

The “{instruction}{text\_to\_classify}” will be at the start of every question/option combination, so I was hoping I could save the hidden state (past\_key\_values) for the “{instruction}{text\_to\_classify}” and then batch process all my question/option combinations. This should save a huge amount of compute, especially when my text\_to\_classify has many tokens.

The sad thing is that I couldn’t figure out how to use past\_key\_values with model() (the \_\_call\_\_ method) for processing batched inputs where

```python
model = AutoModelForCausalLM.from_pretrained("openchat/openchat_3.5")  

```

Advice would be greatly appreciated. Even if it is not possible with transformers as is, I would appreciate those comments too, because then, at least, I know whether I need to get my hands dirty and code something from scratch.