# Using model() instead of model.generate()

**URL:** <https://discuss.huggingface.co/t/using-model-instead-of-model-generate/116329>\
**Category:** 🤗Transformers\
**Created:** [November 8, 2024, 5:15am UTC](https://discuss.huggingface.co/t/using-model-instead-of-model-generate/116329 "2024-11-08T05:15:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vidushee](https://avatars.discourse-cdn.com/v4/letter/v/ecccb3/32.png) [@Vidushee](https://discuss.huggingface.co/u/Vidushee)\
**Post date:** [November 8, 2024, 5:15am UTC](https://discuss.huggingface.co/t/using-model-instead-of-model-generate/116329/1 "2024-11-08T05:15:30Z")

</div>

I am using pixtral-12b and I want to directly use the model() to get the output and their logits, how can it be used to product correct output ? It seems to be producing lots of gibberish when I try to decode the outputs from model().

```auto
        outputs = self.model(**inputs)
        # generated_ids = self.model.generate(**inputs)

        # generate_ids = self.model.generate(**inputs, max_new_tokens=50,do_sample=True,min_p=0.1, temperature=0.9)
        # output = self.processor.batch_decode(generate_ids, skip_special_tokens=True,clean_up_tokenization_spaces=False)
        # print(output[0]) 

        # Get logits for the last token
        logits = outputs.logits[:, -1, :]

        #
        # You might need to use the model's tokenizer to convert logits to text
        generated_ids = logits.argmax(dim=-1)
        generated_text = self.processor.tokenizer.decode(generated_ids[0], skip_special_tokens=True)
        
        print("Generated text:", generated_text)

        # # Initialize an empty list to store all generated token IDs
        # generated_ids = []

        # Get the logits for each token in the generated sequence
        for i in range(outputs.logits.size(1)): # Assuming logits has shape (batch_size, seq_length, vocab_size)
            logits_test = outputs.logits[:, i, :]
            # Get the predicted token id for each step
            token_id = logits_test.argmax(dim=-1)
            generated_ids.append(token_id[0].item()) # Save token id as integer

```

Also using model() takes atleast 2x the memory than model.generate(), I am really confused about using model() directly, can someone refer me to correct docs or give any thoughts on this ?

---

<div class="post-metadata">

**Author:** ![Vidushee](https://avatars.discourse-cdn.com/v4/letter/v/ecccb3/32.png) [@Vidushee](https://discuss.huggingface.co/u/Vidushee)\
**Post date:** [November 8, 2024, 6:03am UTC](https://discuss.huggingface.co/t/using-model-instead-of-model-generate/116329/3 "2024-11-08T06:03:50Z")

</div>

Can you explain a bit more about your answer ? isn’t it just calling the forward pass ?  
And how about this ?

> [@Generate without using the generate method](https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379):
>
> Posting this here for visibility. What if you want to decode the output of a generative seq2seq model (like T5, BART, etc.) yourself, without using the .generate() method? The code example below illustrates this. Suppose that the model is given a long text, for which it needs to generate a summary. We illustrate here how to manually decode the generated ids autoregressively. In each iteration, we add the predicted token id by the model to the decoder\_input\_ids, which are then fed as input to th…

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 8, 2024, 6:17am UTC](https://discuss.huggingface.co/t/using-model-instead-of-model-generate/116329/4 "2024-11-08T06:17:15Z")

</div>

Oh, sorry. I thought it was a normal class. It’s okay, though, because it inherits from the Torch module.

> <https://github.com/huggingface/transformers/issues/14498>
>
> Hello, 
> 
> I am using RL to train Seq2Seq models and I need logits from generate… method ? Since in RL we need to sample from current policy? 
> Does anyone know how I can adapt generate method to get logits ?
> 
> Specifically I am using BART based models. 
> If you guys could give a code snippet or something. That will be a lo of help.
> 
> Please let me know.

---

<div class="post-metadata">

**Author:** ![bwnjnoei](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bwnjnoei/32/40132_2.png) [@bwnjnoei](https://discuss.huggingface.co/u/bwnjnoei)\
**Post date:** [January 30, 2025, 9:42pm UTC](https://discuss.huggingface.co/t/using-model-instead-of-model-generate/116329/5 "2025-01-30T21:42:59Z")

</div>

if you want to use kv cache:  
first time,

```auto
model(input_ids, use_cache=True)

```

then,

```auto
model(next_token_id, use_cache=True, past_key_values=your_past_key_values)

```