# The effect of padding\_side

**URL:** <https://discuss.huggingface.co/t/the-effect-of-padding-side/67188>\
**Category:** 🤗Transformers\
**Created:** [December 27, 2023, 4:32pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188 "2023-12-27T16:32:44Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![zhouzaida](https://avatars.discourse-cdn.com/v4/letter/z/ce7236/32.png) [@zhouzaida](https://discuss.huggingface.co/u/zhouzaida)\
**Post date:** [December 27, 2023, 4:32pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/1 "2023-12-27T16:32:44Z")

</div>

Hello, I have a question about the documentation here ([Generation with LLMs](https://huggingface.co/docs/transformers/llm_tutorial#wrong-padding-side)). Below is a code block, and I’m curious why setting `padding_side` to ‘left’ yields the correct inference result, while setting it to ‘right’ does not work. The `attention_mask` is also passed to the model’s generate method, so theoretically, it should be able to correctly infer the next token.

```python
# The tokenizer initialized above has right-padding active by default: the 1st sequence,
# which is shorter, has padding on the right side. Generation fails to capture the logic.
model_inputs = tokenizer(
    ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
).to("cuda")
generated_ids = model.generate(**model_inputs)
tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

# With left-padding, it works as expected!
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1", padding_side="left")
tokenizer.pad_token = tokenizer.eos_token # Most LLMs don't have a pad token by default
model_inputs = tokenizer(
    ["1, 2, 3", "A, B, C, D, E"], padding=True, return_tensors="pt"
).to("cuda")
generated_ids = model.generate(**model_inputs)
tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

```

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [December 27, 2023, 7:56pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/2 "2023-12-27T19:56:06Z")

</div>

Hi,

This is explained here: [Generation with LLMs](https://huggingface.co/docs/transformers/llm_tutorial#wrong-padding-side).

> LLMs are [decoder-only](https://huggingface.co/learn/nlp-course/chapter1/6?fw=pt) architectures, meaning they continue to iterate on your input prompt. If your inputs do not have the same length, they need to be padded. Since LLMs are not trained to continue from pad tokens, your input needs to be left-padded.

---

<div class="post-metadata">

**Author:** ![zhouzaida](https://avatars.discourse-cdn.com/v4/letter/z/ce7236/32.png) [@zhouzaida](https://discuss.huggingface.co/u/zhouzaida)\
**Post date:** [December 28, 2023, 2:14am UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/3 "2023-12-28T02:14:27Z")

</div>

Hi @nielsr , thanks for your reply. I understand the role of padding, the point that actually confused me was why padding right affects the output of the model, since the attention mask has already been passed in, the padding should be masked out in atten\_weight, and theoretically it shouldn’t have an effect.

---

<div class="post-metadata">

**Author:** ![zhouzaida](https://avatars.discourse-cdn.com/v4/letter/z/ce7236/32.png) [@zhouzaida](https://discuss.huggingface.co/u/zhouzaida)\
**Post date:** [December 28, 2023, 6:30am UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/4 "2023-12-28T06:30:38Z")

</div>

@nielsr thanks for your help. After debugging the code, I found the key to the unexpected behavior (padding\_side=‘right’) is the next\_token comeing from the logit of pad token. I thought it would somehow get the logit of the last non-pad token as the predicted next token, but that’s not actually the case, it simply takes the last token (which could be a pad token).

```python
        while True:
            if synced_gpus:
                # Under synced_gpus the `forward` call must continue until all gpus complete their sequence.
                # The following logic allows an early break if all peers finished generating their sequence
                this_peer_finished_flag = torch.tensor(0.0 if this_peer_finished else 1.0).to(input_ids.device)
                # send 0.0 if we finished, 1.0 otherwise
                dist.all_reduce(this_peer_finished_flag, op=dist.ReduceOp.SUM)
                # did all peers finish? the reduced sum will be 0.0 then
                if this_peer_finished_flag.item() == 0.0:
                    break

            # prepare model inputs
            model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)

            # forward pass to get next token
            outputs = self(
                **model_inputs,
                return_dict=True,
                output_attentions=output_attentions,
                output_hidden_states=output_hidden_states,
            )

            if synced_gpus and this_peer_finished:
                continue # don't waste resources running the code we don't need

            next_token_logits = outputs.logits[:, -1, :]

```

---

<div class="post-metadata">

**Author:** ![DoganK01](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@DoganK01](https://discuss.huggingface.co/u/DoganK01)\
**Post date:** [May 17, 2024, 11:56pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/5 "2024-05-17T23:56:44Z")

</div>

Hi dude, I couldnt quite understand the logic here

And one more thing: I saw this piece of code:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/7/f/7f5e5874b3428578ac8c05c7572d269444bbde4b.png)

decided to pad on left side but with eos token ? Don’t the models automatically stop when they see eos tokens? Shouldn’t there be a problem here?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [May 20, 2024, 9:42am UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/6 "2024-05-20T09:42:01Z")

</div>

Hi,

If models don’t have a padding token set one can use the EOS token as padding token, and pad from the left at inference time.

This is not an issue since the model will then see “\<eos\> \<eos\> \<eos\> (…) hello your name is” =\> then the model is prompted to continue the token “is”, so it will generate several new tokens until it will generate an EOS token.

---

<div class="post-metadata">

**Author:** ![DoganK01](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@DoganK01](https://discuss.huggingface.co/u/DoganK01)\
**Post date:** [May 20, 2024, 9:39pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/7 "2024-05-20T21:39:48Z")

</div>

is it like [EOS, EOS, EOS, Hello, your, name, is, …]? Because in this format, model should stop since it sees the stop token. what is I’m missing ?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [May 21, 2024, 7:00am UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/8 "2024-05-21T07:00:58Z")

</div>

Yes, sorry for Forum was hiding the \<eos\> tokens in my reply 😛

---

<div class="post-metadata">

**Author:** ![DoganK01](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@DoganK01](https://discuss.huggingface.co/u/DoganK01)\
**Post date:** [May 21, 2024, 11:37pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/9 "2024-05-21T23:37:19Z")

</div>

I didnt understand, what is the specific reason to use EOS to do padding it? Why we using EOS? and why left side? isn’t it the case that model stops when it sees the EOS token generated from itsel? (for example [BOS] Hi, how are you? [EOS]). For this example, shouldnt the model just stop since the model generated [EOS] token when the model tokenized “?” ?

It makes sense to use the EOS token when we set the padding side = right. Likewise, we can also use BOS (begin of sentece) tokens for padding, right? And it makes sense when we set the padding side = left. What am I missing?

---

<div class="post-metadata">

**Author:** ![kalpanmukherjee](https://avatars.discourse-cdn.com/v4/letter/k/90ced4/32.png) [@kalpanmukherjee](https://discuss.huggingface.co/u/kalpanmukherjee)\
**Post date:** [June 15, 2024, 6:23pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/10 "2024-06-15T18:23:52Z")

</div>

@DoganK01 from what I understand what happens is the model sees -  
[eos] - nothing to generate  
[eos] [eos] - nothing to generate  
[eos] [eos] hello - generates logits for after hello

hope this clears it up for you!

---

<div class="post-metadata">

**Author:** ![Boltzmachine](https://avatars.discourse-cdn.com/v4/letter/b/46a35a/32.png) [@Boltzmachine](https://discuss.huggingface.co/u/Boltzmachine)\
**Post date:** [September 10, 2024, 4:52pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/11 "2024-09-10T16:52:45Z")

</div>

I cannot understand why huggingface implement like this. Why don’t they extract the last non-pad tokens of each sample?

---

<div class="post-metadata">

**Author:** ![rlee002](https://avatars.discourse-cdn.com/v4/letter/r/85f322/32.png) [@rlee002](https://discuss.huggingface.co/u/rlee002)\
**Post date:** [January 7, 2025, 2:45am UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/12 "2025-01-07T02:45:52Z")

</div>

Adding onto here, I believe this is only for the generation side (inference side) of the model. So for fine-tuning an LLM, do we still keep the right padding or do we follow the same logic as for inference and keep the left padding?

---

<div class="post-metadata">

**Author:** ![MauroExtrac](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mauroextrac/32/38514_2.png) [@MauroExtrac](https://discuss.huggingface.co/u/MauroExtrac)\
**Post date:** [April 17, 2025, 3:55pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/13 "2025-04-17T15:55:22Z")

</div>

Did you ever find out?

---

<div class="post-metadata">

**Author:** ![DoganK01](https://avatars.discourse-cdn.com/v4/letter/d/5fc32e/32.png) [@DoganK01](https://discuss.huggingface.co/u/DoganK01)\
**Post date:** [May 27, 2025, 12:35pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/14 "2025-05-27T12:35:17Z")

</div>

Guys, I figured it out. Since models are decoder-only (autoregressive), its nonsense applying padding on right side. Because model predicts the next token by looking at last as you can figure this out @zhouzaida s last answer in this thread. And about model stopping predicting next token when it sees EOS, its just adjusting it in the code by telling model that it shouldnt focus on padding (EOS) tokens in the beginning and then should skip them. This is what I’ve figured out. But when we say model to skip those padding tokens, it shouldnt have any importance to set pad token to EOS or BOS. I dont have answer for the last one 😃

---

<div class="post-metadata">

**Author:** ![zjysteven](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/zjysteven/32/52239_2.png) [@zjysteven](https://discuss.huggingface.co/u/zjysteven)\
**Post date:** [August 7, 2025, 4:21pm UTC](https://discuss.huggingface.co/t/the-effect-of-padding-side/67188/15 "2025-08-07T16:21:19Z")

</div>

This is indeed the root cause. IMO this can be easily fixed (i.e., by taking the logits of the last non-padding token); not sure why it’s not implemented this way in the first place.