# Batch generation with GPT2 **URL:** \ **Category:** šŸ¤—Transformers\ **Created:** [October 13, 2020, 9:53pm UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517 "2020-10-13T21:53:22Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [October 13, 2020, 9:53pm UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/1 "2020-10-13T21:53:22Z")
How to do batch generation with the GPT2 model? ---
**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\ **Post date:** [October 14, 2020, 1:56pm UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/2 "2020-10-14T13:56:42Z")
Batch generation is now possible for GPT2 in master by leveraging the functionality shown in this PR: [https://github.com/huggingface/transformers/pull/7552?notification\_referrer\_id=MDE4Ok5vdGlmaWNhdGlvblRocmVhZDEyMTMzNzA0MDA6MjM0MjM2MTk%3D#event-3876130796](https://github.com/huggingface/transformers/pull/7552?notification_referrer_id=MDE4Ok5vdGlmaWNhdGlvblRocmVhZDEyMTMzNzA0MDA6MjM0MjM2MTk%3D#event-3876130796) . For more info on how to prepare a GPT2 for batch generation, you can checkout this test: > ---
**Author:** ![ttj](https://avatars.discourse-cdn.com/v4/letter/t/dc4da7/32.png) [@ttj](https://discuss.huggingface.co/u/ttj)\ **Post date:** [October 14, 2020, 2:09pm UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/3 "2020-10-14T14:09:15Z")
Hi I am the author of the PR. You can now do batch generation by calling the same `generate()`. All you need to add is: 1. set `tokenizer.padding_side = "left" ` (probably reset it back later) 2. pass in `attention_mask` to `generate()` Explanation: (see full example in the end) 1. We need `tokenizer.padding_side = "left"` because we will use the logits of the right-most token to predict the next token, so the padding should be on the left. 2. This what this PR added. Here is a summary: GPT-2 uses absolute positional embedding (`position_ids`), before this change, no `position_ids ` is passed in to the model, and the model automatically generates the embeddings from 0 to n, even if there is padding (e.g. when input is a batch). Example: tokens=` a b c` -\> position\_ids=`0 1 2 3 4`, what we expect is `x x 0 1 2` (`x` means donā€™t case) This PR adds positional embedding in `prepare_inputs_for_generation()`, which is called in `generate()`, by calculating them using `attention_mask`, and thatā€™s why you need to pass it in. You can find a full example in [PR](https://github.com/huggingface/transformers/pull/7552#issue-497255933). ---
**Author:** ![Knight625](https://avatars.discourse-cdn.com/v4/letter/k/2acd7d/32.png) [@Knight625](https://discuss.huggingface.co/u/Knight625)\ **Post date:** [November 12, 2021, 7:00pm UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/5 "2021-11-12T19:00:51Z")
Hi, there. Thanks for your work to support batch inference in GPT2. However, I still have one confusion, which may need your help. Thanks in advance! If I wanna pass the ā€œpast\_key\_valuesā€, how should I process the position\_ids and attention mask? Supposing the length of my past\_key\_values is 2, the padded input is just like your example: , , a, b, c. Should I change the attention mask from 0, 0, 1, 1, 1 to 1, 1, 0, 0, 1, 1, 1, where the first double ā€œ1ā€ refers. to the past\_key\_values. Thanks a lot! ---
**Author:** ![deathcrush](https://avatars.discourse-cdn.com/v4/letter/d/58f4c7/32.png) [@deathcrush](https://discuss.huggingface.co/u/deathcrush)\ **Post date:** [February 28, 2022, 10:41am UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/6 "2022-02-28T10:41:55Z")
@patrickvonplaten @ttj I think this is a good question! Could we discuss on how to do batch inference with `past_key_values`? ---
**Author:** ![sasha](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sasha/32/5131_2.png) [@sasha](https://discuss.huggingface.co/u/sasha)\ **Post date:** [July 7, 2022, 5:09pm UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/7 "2022-07-07T17:09:09Z")
Is it possible to have variable `max_gen_length`? depending on the length of the input sequence, for instance? (e.g. `max_gen_length = len(tokenizer.tokenize(input_seq) + 20)`? ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [July 7, 2022, 5:48pm UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/8 "2022-07-07T17:48:56Z")
It looks like you are looking for [`max_new_tokens`](https://huggingface.co/docs/transformers/main_classes/text_generation#transformers.generation_utils.GenerationMixin.generate.max_new_tokens)? ---
**Author:** ![milkBottle](https://avatars.discourse-cdn.com/v4/letter/m/73ab20/32.png) [@milkBottle](https://discuss.huggingface.co/u/milkBottle)\ **Post date:** [September 29, 2022, 4:10am UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/9 "2022-09-29T04:10:22Z")
hi, Iā€™m using the input parameter ā€œpast\_key\_valuesā€ to train a gpt model. So I wonder when doing batch generation in this way, if I pass ā€œpast\_key\_valuesā€ to model through the parameter ā€œmodel\_kwargsā€, whether the generation method will work as expected? Thx! ---
**Author:** ![lqtrung](https://avatars.discourse-cdn.com/v4/letter/l/91b2a8/32.png) [@lqtrung](https://discuss.huggingface.co/u/lqtrung)\ **Post date:** [February 10, 2023, 7:06am UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/10 "2023-02-10T07:06:25Z")
Hello, Correct me if Iā€™m wrong but for GPT2, during training mode, when `padding_side='left'`, position\_ids should be passed correctly or else the position\_ids is created as if right padding is used ā€¦ making it inconsistent, right ? Iā€™m referring to this line where the position\_ids is created if it is not passed ā€¦ [transformers/modeling\_gpt2.py at main Ā· huggingface/transformers (github.com)](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_gpt2.py#L801) Thank you ! ---
**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\ **Post date:** [February 15, 2023, 10:49am UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/11 "2023-02-15T10:49:36Z")
Hey @lqtrung šŸ‘‹ The `position_ids` donā€™t need to be passed, as long as the right `attention_mask` is. `prepare_inputs_for_generation` (see [here](https://github.com/huggingface/transformers/blob/e3d832ff87c6ec997125deaa4f1b239db8f9e613/src/transformers/models/gpt2/modeling_gpt2.py#L1005)) takes care of that for you šŸ˜ƒ ---
**Author:** ![lqtrung](https://avatars.discourse-cdn.com/v4/letter/l/91b2a8/32.png) [@lqtrung](https://discuss.huggingface.co/u/lqtrung)\ **Post date:** [February 16, 2023, 3:03am UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/12 "2023-02-16T03:03:24Z")
> [@lqtrung](#): > > Hello, > > Correct me if Iā€™m wrong but for GPT2, during training mode, when `padding_side='left'`, position\_ids should be passed correctly or else the position\_ids is created as if right padding is used ā€¦ making it inconsistent, right ? > > Iā€™m referring to this line where the position\_ids is created if it is not passed ā€¦ > [transformers/modeling\_gpt2.py at main Ā· huggingface/transformers (github.com)](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gpt2/modeling_gpt2.py#L801) > > Thank you ! Hi @joaogante , thank you for the response. I believe that the position\_ids is properly prepared during generation as you said because the prepare\_inputs\_for\_generation is called ā€¦ But my question is about during training where that function is not called and the gpt2 modeling script does not compute position\_ids based on the attention mask (so it is not correct when ā€˜leftā€™ padding is used ā€¦) So Iā€™m not sure about the recommended practice: 1. Is ā€˜rightā€™ padding always used during training ā€¦ and ā€˜leftā€™ padding is only used during batch generation ? 2. Or the training and generation should have the same padding scheme and in this case the gpt2 modeling script should handle the position\_ids better ? ---
**Author:** ![joaogante](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/joaogante/32/20106_2.png) [@joaogante](https://discuss.huggingface.co/u/joaogante)\ **Post date:** [February 16, 2023, 10:54am UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/13 "2023-02-16T10:54:35Z")
@lqtrung what you described as option 1. (right padding during training, left padding during inference) is the way to go. You can also always pass `position_ids`, but the settings above get you the correct results without passing them. A caveat here is that you never want GPT2 to generate after its pad token (note: GPT2 doesnā€™t have a pad token, but it is common to set pad token = eos token), even if you pass the correct `position_ids`. GPT2 was not trained for that case, and the results will be gibberish ā€“ right padding will often get you in this situation. A good resource to reason about this is the [illustrated GPT2](https://jalammar.github.io/illustrated-gpt2/) šŸ™‚ ---
**Author:** ![marsggbo](https://avatars.discourse-cdn.com/v4/letter/m/c0e974/32.png) [@marsggbo](https://discuss.huggingface.co/u/marsggbo)\ **Post date:** [January 16, 2024, 2:28am UTC](https://discuss.huggingface.co/t/batch-generation-with-gpt2/1517/14 "2024-01-16T02:28:45Z")
The code is designed to pad all tokens to the same length, equivalent to the maximum length of one input. However, itā€™s important to note that during inference, the output lengths of different inputs can vary. After testing the code with nine sentences, where the input lengths range from 32 to 1140, it was observed that `model.generate(inputs_padded)` completed after only one forward pass (decoding). This suggests that the model didnā€™t perform decoding correctly. An additional attempt was made using `model.generate(inputs_padded, max_new_tokens=64)`. However, this resulted in an error related to CUDA. This is probably because some sentences has completed generation while others have not. Any suggestions to solve these problems?