# How to implement generate function for seperate encoder decoder T5 model?

**URL:** <https://discuss.huggingface.co/t/how-to-implement-generate-function-for-seperate-encoder-decoder-t5-model/14582>\
**Category:** Models\
**Created:** [February 10, 2022, 1:43pm UTC](https://discuss.huggingface.co/t/how-to-implement-generate-function-for-seperate-encoder-decoder-t5-model/14582 "2022-02-10T13:43:42Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![vikasojha](https://avatars.discourse-cdn.com/v4/letter/v/43a26b/32.png) [@vikasojha](https://discuss.huggingface.co/u/vikasojha)\
**Post date:** [February 10, 2022, 1:43pm UTC](https://discuss.huggingface.co/t/how-to-implement-generate-function-for-seperate-encoder-decoder-t5-model/14582/1 "2022-02-10T13:43:42Z")

</div>

I was working on optimizing the T5 model. The version of the transformer I am using is 4.8. For optimization, I separated the model into encoder and decoder with LM(Language modeling) head. Earlier for generation, I just passed input\_ids, attention\_mask,max\_length, and num\_beams. But now since the model is separated into encoder and decoder I can’t use generate function. But I am able to generate logits using the below code:

encoder\_last\_hidden\_state = t5\_trt\_encoder(input\_ids=input\_ids)  
outputs = t5\_trt\_decoder(input\_ids, encoder\_last\_hidden\_state)

But how can I use these logits to generate sequences?. I mean I am unclear how can I use the encoder and decoder to write a class which supports generate function?

When I am prediction in this way:

# Generate sequence for an input

from transformers.generation\_stopping\_criteria import (  
MaxLengthCriteria,  
StoppingCriteriaList,  
)

max\_length = 64

decoder\_input\_ids = torch.full(  
(1, 1), tokenizer.convert\_tokens\_to\_ids(tokenizer.pad\_token), dtype=torch.int32  
)  
encoder\_last\_hidden\_state = t5\_trt\_encoder(input\_ids=input\_ids)

outputs = t5\_trt\_decoder.greedy\_search(  
input\_ids=decoder\_input\_ids,  
encoder\_hidden\_states=encoder\_last\_hidden\_state,  
stopping\_criteria = StoppingCriteriaList([MaxLengthCriteria(max\_length)])  
)  
print(tokenizer.decode(outputs[0], skip\_special\_tokens=True))

The output is blank.  
The value of outputs[0] is:  
tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,  
0, 0, 0, 0, 0, 0, 0, 0])

The model I am trying to optimize is valhalla/t5-small-qa-qg-hl model from [GitHub - patil-suraj/question\_generation: Neural question generation using transformers](https://github.com/patil-suraj/question_generation) repo and I am using using trt\_torch\_encoder and trt\_torch\_decoder from this notebook  
[https://github.com/NVIDIA/TensorRT/blob/main/demo/HuggingFace/notebooks/t5.ipynb](https://github.com/NVIDIA/TensorRT/blob/main/demo/HuggingFace/notebooks/t5.ipynb)