# Batch generation Llama 3 Instruct | Tokenizer has no padding token

**URL:** <https://discuss.huggingface.co/t/batch-generation-llama-3-instruct-tokenizer-has-no-padding-token/168043>\
**Category:** 🤗Transformers\
**Created:** [September 2, 2025, 8:07pm UTC](https://discuss.huggingface.co/t/batch-generation-llama-3-instruct-tokenizer-has-no-padding-token/168043 "2025-09-02T20:07:06Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](https://discuss.huggingface.co/u/samirchar)\
**Post date:** [September 2, 2025, 8:07pm UTC](https://discuss.huggingface.co/t/batch-generation-llama-3-instruct-tokenizer-has-no-padding-token/168043/1 "2025-09-02T20:07:06Z")

</div>

Hello everyone,

What is the best way of using a model like Llama 3.1 ( [meta-llama/Llama-3.1-8B-Instruct · Hugging Face](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) ) with AutoModel, AutoTokenizer, and template chat (I can’t use pipelines for my use case) **for batch generation** and eventually also using DDP.

This works for a single conversation:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

messages = [
    {
        "role": "system",
        "content": "You are a pirate chatbot who always responds in pirate speak!",
    },
    {"role": "user", "content": "Who are you?"},
]

input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>"),
]

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)

response = outputs[0][input_ids.shape[-1] :]
print(tokenizer.decode(response, skip_special_tokens=True))

```

For multiple conversations and batch decoding, do I just need to apply the chat template with padding = True? When I try that, I get the error “Asking to pad but the tokenizer does not have a padding token”

---

<div class="post-metadata">

**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](https://discuss.huggingface.co/u/samirchar)\
**Post date:** [September 2, 2025, 8:43pm UTC](https://discuss.huggingface.co/t/batch-generation-llama-3-instruct-tokenizer-has-no-padding-token/168043/2 "2025-09-02T20:43:55Z")

</div>

Actually, could this be the solution?

1. Set padding to left

2. Set pad token to eos token

3. In generate set pad token id to eos token id

4. Use tokenizer.batch\_decode

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")
tokenizer.pad_token = tokenizer.eos_token
tokenizer.pad_token_id = tokenizer.eos_token_id

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    [
        {
            "role": "system",
            "content": "You are a pirate chatbot who always responds in pirate speak!",
        },
        {"role": "user", "content": "Who are you?"},
    ],
    [
        {
            "role": "system",
            "content": "You are a pirate chatbot who always responds in pirate speak!",
        },
        {"role": "user", "content": "How old are you?"},
    ],
]

input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", padding=True
).to(model.device)

terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>"),
]

outputs = model.generate(
    input_ids,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
    pad_token_id=tokenizer.eos_token_id,
)
tokenizer.batch_decode(outputs, skip_special_tokens=True)

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [September 3, 2025, 3:34am UTC](https://discuss.huggingface.co/t/batch-generation-llama-3-instruct-tokenizer-has-no-padding-token/168043/3 "2025-09-03T03:34:59Z")

</div>

I think that’s correct. If anything else to add, maybe `return_dict=True` or something.

```py
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "meta-llama/Llama-3.1-8B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, padding_side="left")
tokenizer.pad_token = tokenizer.eos_token
tokenizer.pad_token_id = tokenizer.eos_token_id # inference-safe

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

messages = [
    [
        {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
        {"role": "user", "content": "Who are you?"},
    ],
    [
        {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
        {"role": "user", "content": "How old are you?"},
    ],
]

# Return a BatchEncoding with input_ids **and** attention_mask, already padded on the left
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True, # explicit
    return_tensors="pt",
    return_dict=True, # crucial for batched generate
    padding=True,
).to(model.device)

terminators = [tokenizer.eos_token_id, tokenizer.convert_tokens_to_ids("<|eot_id|>")]

outputs = model.generate(
    **inputs, # pass dict, not a single tensor
    max_new_tokens=256,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
    eos_token_id=terminators, # stop on EOS or EOT
    pad_token_id=tokenizer.eos_token_id,
)

# Drop the prompt, then decode the new tokens only
new_tokens = outputs[:, inputs["input_ids"].shape[1]:]
texts = tokenizer.batch_decode(new_tokens, skip_special_tokens=True)

```

---

<div class="post-metadata">

**Author:** ![samirchar](https://avatars.discourse-cdn.com/v4/letter/s/c2a13f/32.png) [@samirchar](https://discuss.huggingface.co/u/samirchar)\
**Post date:** [September 3, 2025, 11:04am UTC](https://discuss.huggingface.co/t/batch-generation-llama-3-instruct-tokenizer-has-no-padding-token/168043/4 "2025-09-03T11:04:36Z")

</div>

That’s awesome, thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [September 3, 2025, 11:05pm UTC](https://discuss.huggingface.co/t/batch-generation-llama-3-instruct-tokenizer-has-no-padding-token/168043/5 "2025-09-03T23:05:14Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.