# Variable length batch decoding **URL:** \ **Category:** 🤗Transformers\ **Created:** [October 4, 2020, 4:11pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359 "2020-10-04T16:11:28Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\ **Post date:** [October 4, 2020, 4:11pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/1 "2020-10-04T16:11:28Z")
Hi All, Just want to know, is there any way to batch decode variable length sentences. For example [S1, S2] , where S1 has 5 words abd S2 has 10 words . Can we decode it using GPT2 , BERT etc? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 4, 2020, 4:16pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/2 "2020-10-04T16:16:06Z")
Hi @s4sarath What do you mean by decoding, as in decoding the generated tokens by GPT2 or making predictions on a batch of sequences ? ---
**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\ **Post date:** [October 4, 2020, 4:31pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/3 "2020-10-04T16:31:34Z")
By Decoding I mean, generate sequence of tokens. @valhalla . ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 4, 2020, 4:36pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/4 "2020-10-04T16:36:36Z")
All `tokenizers` offer this functionality, just pass the list of seqs to it ```python tokens = tokenizer([s1, s2])["input_ids"] ``` by default it’ll pad all the seqs to the maximum length in the batch if they are of different length. You can find more detailed info in this [guide](https://huggingface.co/transformers/preprocessing.html) ---
**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\ **Post date:** [October 4, 2020, 4:41pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/5 "2020-10-04T16:41:18Z")
@valhalla thanks for this. I have seen that. But when I try the pad token in gpt2 it dint work as expected. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 4, 2020, 4:47pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/6 "2020-10-04T16:47:20Z")
There is no pad token for GPT, you can manually set the `eos` token as `pad` token using ```python tokenizer.pad_token = tokenizer.eos_token ``` ---
**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\ **Post date:** [October 5, 2020, 3:03am UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/7 "2020-10-05T03:03:41Z")
Hi @valhalla . Thanks for suggestion . I have done as per you say. This is the same thing I have tried before. But as you see in screenshot, the results of variable batch sentences does not produce correct results. ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/a/a1bc2382a6f47208cca94d50e08f1fbae7aad148.png) ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 5, 2020, 9:23am UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/8 "2020-10-05T09:23:25Z")
As this is auto regressive model which predicts next token based on previous tokens, it might not generate correct tokens when there are eos in the text. I thought you were asking about batching at training time. Sorry about the misleading answer. Right now `generate` does not support batched generation for gpt2. Pinging @lysandre ---
**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\ **Post date:** [October 5, 2020, 11:53am UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/9 "2020-10-05T11:53:33Z")
No problem @valhalla . Appreciate your response. I have implemented this feature locally. Off topic, would like to know what is your thoughts on this. > [@Why Tensorflow Models are way slower than Pytorch models, for autoregressive modeling?](https://discuss.huggingface.co/t/why-tensorflow-models-are-way-slower-than-pytorch-models-for-autoregressive-modeling/1332): > > Hi, I was experimenting with many models include GPT2, T5 etc. But it seems like Tensorflow models are too slow for same type of generation comparing to Tensorflow, whether it is greedy, beam etc . Any specific reasons for this? Thanks ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 5, 2020, 12:28pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/10 "2020-10-05T12:28:12Z")
Not a TF user 🙂 Great, can you share your fix if possible, lots of other people are interested in batched prediction for GPT. ---
**Author:** ![s4sarath](https://avatars.discourse-cdn.com/v4/letter/s/bc79bd/32.png) [@s4sarath](https://discuss.huggingface.co/u/s4sarath)\ **Post date:** [October 5, 2020, 1:18pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/11 "2020-10-05T13:18:44Z")
I have implemented in TF2.0 . I had to make quite a bunch of changes to make it work. Will share it. ---
**Author:** ![fzyzcjy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fzyzcjy/32/25229_2.png) [@fzyzcjy](https://discuss.huggingface.co/u/fzyzcjy)\ **Post date:** [March 31, 2024, 1:27pm UTC](https://discuss.huggingface.co/t/variable-length-batch-decoding/1359/12 "2024-03-31T13:27:47Z")
Hi, is there any updates? Thanks!