# Big \`generate()\` refactor **URL:** \ **Category:** šŸ¤—Transformers\ **Created:** [November 2, 2020, 12:49pm UTC](https://discuss.huggingface.co/t/big-generate-refactor/1857 "2020-11-02T12:49:48Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [November 2, 2020, 12:49pm UTC](https://discuss.huggingface.co/t/big-generate-refactor/1857/1 "2020-11-02T12:49:48Z")
Soon we will merge a big refactor of the `generate()` method: [https://github.com/huggingface/transformers/pull/6949](https://github.com/huggingface/transformers/pull/6949) All transformer models that have a language model head rely on the `generate()` method, _e.g._ - Bart, T5, Marian, ProphetNet for summarization, translation, ā€¦ - GPT2, XLNet for open-end text generation The `generate()` method has become hard to read and more and quite difficult to maintain. The goal of the refactor is thus as follows: 1. Remove the `_no_beam_search_generation` and `_beam_search_generation` methods and replace them with the four different generation methods corresponding to the four different types of generation: `greedy_search` (corresponds to num\_beams = 1, do\_sample=False), `sample` (corresponds to num\_beams = 1, do\_sample=True), `beam_search` (corresponds to num\_beams \> 1, do\_sample = False), and `beam_sample` (corresponds to num\_beams \> 1, do\_sample = True). The following philosophy has been adapted here: the original `generate()` method is kept at 100% backwards compatibility (ping me on github under @patrickvonplaten, if your specific use case has nevertheless been broken by this PR). In addition each of the four specific generation methods can be used directly. The specific methods are as ā€œbare-boneā€ as possible meaning that they donā€™t contain any magic pre-processing of input tensors. _E.g._ `generate()` automatically runs the `input_ids` through the encoder if model is encoder-decoder model vs. for each specific generate method they have to be added as `encoder_outputs` to the `model_kwargs`; `input_ids` are automatically created in case they are empty or automatically expanded for `num_beams` in `generate()` vs. this has to be done manually for each specific generate method. The reason behind this design is that it will give the user much more flexibility for specific use cases of `generate()` and improves maintainability and readability. The user should not be limited in any way when directly using the specific generate methods. It should therefore pave the way to allow backprop through generate, make beam generation much easier, makes it easier to write ā€œhigher levelā€ generate functions as is done for RAGā€¦ For more information on this design please read the docs, look into the examples of `greedy_search`, `sample`, `beam_search` and `beam_sample`. 2. All of the generate parameters that can be used to tweak the logits distribution for better generation results, _e.g._ `no_repeat_ngram_size`, `min_length`, ā€¦ are now defined as separate classes that are added to a `LogitsProcessorList`. This has the following advantages: a) better readability b) much easier to test these functions c) easier to add new logits distribution warpers. A huge thanks goes to [https://github.com/turtlesoupy](https://github.com/turtlesoupy) who has had the original idea of this design in this PR: [https://github.com/huggingface/transformers/pull/5420](https://github.com/huggingface/transformers/pull/5420) 3. Move all `beam_search` relevant code into its own generation\_beam\_search.py file and speed up beam search. Beam search has gained more and more in importance thanks to many new and improved seq2seq models. This PR moves the very difficult to understand beam search code into its own file and makes sure that the `beam_search` generate function is easier to understand this way. Additionally, all Python List operations are now replaced by torch.tensor operations which led to a 5-10% speed up for `beam_search` generation. This change improves speed, readability and also maintainability. New beam search algorithms, such as [https://arxiv.org/abs/2007.03909](https://arxiv.org/abs/2007.03909) , should now be easier to add to the library. 4. Tests have been refactored and new more aggressive tests have been added. The tests can also be very helpful to understand how each of the methods work exactly. Check out `test_generation_utils.py`, `test_generation_beam_search` and `test_generation_logits_process`. 5. More docstring especially on beam search and logits processors to make generate more accessible and understandable to the user. TODO: - Do the same refactor for TF - Check possibility of carrying gradients through generate - Add `GenerationOutputs` similar to `ModelOutputs` that allows to return attention outputs and hidden states ---
**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](https://discuss.huggingface.co/u/Jung)\ **Post date:** [November 3, 2020, 3:07am UTC](https://discuss.huggingface.co/t/big-generate-refactor/1857/2 "2020-11-03T03:07:02Z")
Hi Patrick @patrickvonplaten, Thanks for your great work!! I have a question. I now almost finish translating [TFDPR](https://github.com/huggingface/transformers/issues/8171) , and currently also working on TFRag (I already made `TFRagModel` works like Pytorch, and I am investigating `TFRagTokenForGeneration` right now) . Since the current code of `RagTokenForGeneration` involves with both `_no_beam_search_generation` and `_beam_search_generation` , regarding the Refactor, I think I should just go ahead with the current version, and fix them later for refactor. Or would you suggest that I wait for Pytorch Refactor, or wait for both Pytorch & TF Refactor ? ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [November 3, 2020, 9:35am UTC](https://discuss.huggingface.co/t/big-generate-refactor/1857/3 "2020-11-03T09:35:52Z")
Thatā€™s amazing to hear that you are working on the TFRagModel ! do you already have a PR on this? Iā€™d definitely say that you should go ahead with `_no_beam_search_generation` and `_beam_search_generation` in TF (itā€™ll take probably at least ~2,3 weeks until I start the TF refactor and \> a month until I merge it). Feel free to tag me on the TFRagModel šŸ™‚ ---
**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](https://discuss.huggingface.co/u/Jung)\ **Post date:** [November 3, 2020, 10:58am UTC](https://discuss.huggingface.co/t/big-generate-refactor/1857/4 "2020-11-03T10:58:44Z")
Hi Patrick, thank you for your suggestion. as Rag uses DPRā€™s pretrained weights as a submodule, so I made a TFDPR PR first. Now itā€™s around 25% progress on TFRag ā€¦ I hope I can make a TFRag PR before EMNLP , and I will seek your suggestions on TFRag for sure šŸ˜ƒ Just in case you are interested in, my current draft with TFRagModel is [here](https://colab.research.google.com/drive/1mcVh1XAm-FSuY5CvyqHuoTr32CTtSaRx?usp=sharing) ---
**Author:** ![Buckeyes2019](https://avatars.discourse-cdn.com/v4/letter/b/7c8e57/32.png) [@Buckeyes2019](https://discuss.huggingface.co/u/Buckeyes2019)\ **Post date:** [December 22, 2020, 2:32pm UTC](https://discuss.huggingface.co/t/big-generate-refactor/1857/5 "2020-12-22T14:32:01Z")
Hello ā€“ on a hopefully related note, I saw in a recent Huggingface email that you had implemented Diverse Beam Search. I was wondering, how is this applied and how can I use it? I have been using beam search with sampling in a summarization pipeline (via Bart-Large-cnn), and I was wondering how I can add more diversity to each beam generated? Thanks! ---
**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](https://discuss.huggingface.co/u/Jung)\ **Post date:** [December 23, 2020, 2:34am UTC](https://discuss.huggingface.co/t/big-generate-refactor/1857/6 "2020-12-23T02:34:49Z")
Hi you can use the new diverse beam generation easily by just adding the two new parameters into `generate()`. Two new parameters on diverse beam are `num_beam_group` and `diversity_penalty` where `num_beam_group <= num_beams` **Example** ``` outputs = model.generate(input_ids, min_length=1, max_length=200, no_repeat_ngram_size=3, num_beams=5, early_stopping=True, do_sample=False, num_return_sequences=5, num_beam_groups=5, diversity_penalty=2.0 ) ``` ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [December 25, 2020, 10:25pm UTC](https://discuss.huggingface.co/t/big-generate-refactor/1857/7 "2020-12-25T22:25:17Z")
It only works with `do_sample=False` however ---
**Author:** ![cahya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cahya/32/48725_2.png) [@cahya](https://discuss.huggingface.co/u/cahya)\ **Post date:** [November 26, 2021, 6:05am UTC](https://discuss.huggingface.co/t/big-generate-refactor/1857/8 "2021-11-26T06:05:34Z")
I just knew about diverse beam search, I have to try it for my abstractive text summarizer but unfortunately it works mostly extractive. I hope this diverse beam search could help to create abstractive summary little bit