# Generate without using the generate method **URL:** \ **Category:** Intermediate\ **Created:** [November 5, 2021, 9:30am UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379 "2021-11-05T09:30:37Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\ **Post date:** [November 5, 2021, 9:30am UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379/1 "2021-11-05T09:30:37Z")
Posting this here for visibility. What if you want to decode the output of a generative seq2seq model (like T5, BART, etc.) yourself, without using the `.generate()` method? The code example below illustrates this. Suppose that the model is given a long text, for which it needs to generate a summary. We illustrate here how to manually decode the generated ids autoregressively. In each iteration, we add the predicted token id by the model to the `decoder_input_ids`, which are then fed as input to the next time step. At the beginning, we only feed the decoder\_start\_token\_id to the decoder of the model. ```auto from transformers import BartTokenizer, BartForConditionalGeneration import torch model_name = "sshleifer/distilbart-cnn-6-6" tokenizer = BartTokenizer.from_pretrained(model_name) model = BartForConditionalGeneration.from_pretrained(model_name) text = """The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct.""" input_ids = tokenizer(text, return_tensors="pt").input_ids decoder_input_ids = [model.config.decoder_start_token_id] predicted_ids = [] for i in range(20): outputs = model(input_ids=input_ids, decoder_input_ids=torch.tensor([decoder_input_ids])) logits = outputs.logits[:,i,:] # perform argmax on the last dimension (i.e. greedy decoding) predicted_id = logits.argmax(-1) predicted_ids.append(predicted_id.item()) print(tokenizer.decode([predicted_id.squeeze()])) # add predicted id to decoder_input_ids decoder_input_ids = decoder_input_ids + [predicted_id] ``` This will print: ```auto The E iff el Tower is 324 metres ( 1 , 06 3 ft ) tall , about the same ``` The final result can also be printed using `print(tokenizer.decode(predicted_ids))`: ```auto The Eiffel Tower is 324 metres (1,063 ft) tall, about the same ``` Note that we’ve only done 20 time steps here. Normally, one continues until the model generates the EOS (end of sequence) token, which for BART is ``. ---
**Author:** ![marshmellow77](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/marshmellow77/32/3279_2.png) [@marshmellow77](https://discuss.huggingface.co/u/marshmellow77)\ **Post date:** [November 26, 2021, 12:56pm UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379/2 "2021-11-26T12:56:43Z")
Hi Niels, thanks for sharing the code. Would you mind also sharing some examples of situations in which you would prefer not to use the `.generate()` method? ---
**Author:** ![rupeshpoojary97](https://avatars.discourse-cdn.com/v4/letter/r/d26b3c/32.png) [@rupeshpoojary97](https://discuss.huggingface.co/u/rupeshpoojary97)\ **Post date:** [April 21, 2022, 10:11am UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379/3 "2022-04-21T10:11:52Z")
If you are deploying your model on triton server and you are inferencing through triton client there is no generate method for your help, I used this method to decode my output through the model. ---
**Author:** ![ashutoshml](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ashutoshml/32/18924_2.png) [@ashutoshml](https://discuss.huggingface.co/u/ashutoshml)\ **Post date:** [December 13, 2022, 5:02am UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379/4 "2022-12-13T05:02:52Z")
Let us suppose we want to restrict our vocabulary to some specific set of tokens (that changes dynamically with each time step). What is the best way of incorporating that? Other than decoding each token individually? ---
**Author:** ![Ujan](https://avatars.discourse-cdn.com/v4/letter/u/6de8d8/32.png) [@Ujan](https://discuss.huggingface.co/u/Ujan)\ **Post date:** [August 1, 2023, 6:18am UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379/5 "2023-08-01T06:18:11Z")
Thanks for the post. I’m finding this to be much slower than the generate( ) function for my use case (whisper model). Is this expected? ---
**Author:** ![yblainm](https://avatars.discourse-cdn.com/v4/letter/y/4bbf92/32.png) [@yblainm](https://discuss.huggingface.co/u/yblainm)\ **Post date:** [September 2, 2023, 2:22pm UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379/6 "2023-09-02T14:22:22Z")
That might be because this doesn’t cache the hidden states when generating, if I understand correctly. You would need to keep past\_key\_values or something like that by making sure use\_cache is True in your model config. Otherwise in the above snippet you’re re-computing the entire past sequence every time you want a next token, despite the fact that causal attention means all the past hidden states are constant. ---
**Author:** ![espoir](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/espoir/32/5153_2.png) [@espoir](https://discuss.huggingface.co/u/espoir)\ **Post date:** [September 15, 2023, 8:02am UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379/7 "2023-09-15T08:02:10Z")
This may help a lot. What if the decoding is using beam search? ---
**Author:** ![sreebee11](https://avatars.discourse-cdn.com/v4/letter/s/ecd19e/32.png) [@sreebee11](https://discuss.huggingface.co/u/sreebee11)\ **Post date:** [September 6, 2024, 12:40am UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379/8 "2024-09-06T00:40:42Z")
Hi! Your reply is very helpful on this. I am trying to do something similar but with the more recent VLMs like Qwen-VL, Llava, etc. I am trying to understand a few things in this: 1. In the output logits, what does the dimension of the sequence\_length denote? As in, what is that sequence? Is it only the input or the output or is it the whole sequence? 2. In your code, in the loop, why do we do output.logits[:, i, :], ie, why do we slice using i? ---
**Author:** ![L-jasmine](https://avatars.discourse-cdn.com/v4/letter/l/82dd89/32.png) [@L-jasmine](https://discuss.huggingface.co/u/L-jasmine)\ **Post date:** [January 17, 2025, 2:36pm UTC](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379/9 "2025-01-17T14:36:21Z")
Thank you so much for your post