# Decoder only model - how to have it not include the prompt in its output?

**URL:** <https://discuss.huggingface.co/t/decoder-only-model-how-to-have-it-not-include-the-prompt-in-its-output/86759>\
**Category:** 🤗Transformers\
**Created:** [May 15, 2024, 3:38am UTC](https://discuss.huggingface.co/t/decoder-only-model-how-to-have-it-not-include-the-prompt-in-its-output/86759 "2024-05-15T03:38:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![amperie](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@amperie](https://discuss.huggingface.co/u/amperie)\
**Post date:** [May 15, 2024, 3:38am UTC](https://discuss.huggingface.co/t/decoder-only-model-how-to-have-it-not-include-the-prompt-in-its-output/86759/1 "2024-05-15T03:38:06Z")

</div>

Using decoder only models it seems that they always output the prompt first with the answer after.

Is there any way to supress the model from spitting out the prompt first and just have it output the answer?

T5, which is an encoder-decoder model has no problem doing this and giving out only the answer.

Possible? Am I missing something obvious?

An example of what I’m talking about is below. (I shortened the context). Every time, it outputs all of the input, with the Yes/No at the very end.

Answer the following question with a yes or no with the context given. Answer with only one word - yes or no. Question: Do obese patients with idiopathic pulmonary fibrosis have a higher 90-day mortality risk with bilateral lung transplantation?  
Context: (‘Obese patients with idiopathic pulmonary fibrosis (IPF) … BMI of 18.5 to 30 kg/m(2).’)  
Remember to only answer with one word - yes or no.

Yes

Thanks in advance!

---

<div class="post-metadata">

**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\
**Post date:** [May 15, 2024, 8:00am UTC](https://discuss.huggingface.co/t/decoder-only-model-how-to-have-it-not-include-the-prompt-in-its-output/86759/2 "2024-05-15T08:00:19Z")

</div>

Hi!

If you want only the generated text for for decoder-only models, you can crop the output manually as follows:

```python
input_length = inputs.input_ids.shape[-1]
output = model.generate(**inputs)
output_only_answer = output[:, input_length:]

```

---

<div class="post-metadata">

**Author:** ![amperie](https://avatars.discourse-cdn.com/v4/letter/a/5f8ce5/32.png) [@amperie](https://discuss.huggingface.co/u/amperie)\
**Post date:** [May 15, 2024, 10:56pm UTC](https://discuss.huggingface.co/t/decoder-only-model-how-to-have-it-not-include-the-prompt-in-its-output/86759/3 "2024-05-15T22:56:47Z")

</div>

Ok yeah I guess there’s no way to supress it since this is what decoders do, put one word at a time at the end of the input… thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [May 16, 2024, 10:57am UTC](https://discuss.huggingface.co/t/decoder-only-model-how-to-have-it-not-include-the-prompt-in-its-output/86759/4 "2024-05-16T10:57:46Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.