# Potential bug with beam search + eos\_token\_id

**URL:** <https://discuss.huggingface.co/t/potential-bug-with-beam-search-eos-token-id/35862>\
**Category:** 🤗Transformers\
**Created:** [April 7, 2023, 7:53am UTC](https://discuss.huggingface.co/t/potential-bug-with-beam-search-eos-token-id/35862 "2023-04-07T07:53:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![zl7](https://avatars.discourse-cdn.com/v4/letter/z/f475e1/32.png) [@zl7](https://discuss.huggingface.co/u/zl7)\
**Post date:** [April 7, 2023, 7:53am UTC](https://discuss.huggingface.co/t/potential-bug-with-beam-search-eos-token-id/35862/1 "2023-04-07T07:53:34Z")

</div>

I want the generation to stop at certain eos\_token\_ids (e.g. “\n”, “.”). I keep failing at [transformers/beam\_search.py at c1f85598eb637bb7b21f8c3b15886aaed40ca42e · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/c1f85598eb637bb7b21f8c3b15886aaed40ca42e/src/transformers/generation/beam_search.py#L283) because there are not enough non-eos-tokens in `next_tokens`.

1. Could someone explain to me what does “Make sure {next\_tokens[batch\_idx]} are corrected.” mean?
2. I tried specifying `early_stopping=True` in my GenerationConfig but it doesn’t help.

I think in this case, if all beams are saying “stop”, we should just stop? Or keep track of the beams that are not stopping. Is there any way to get around this error? In fact, if the code just executes one more round, with `early_stopping=True` [transformers/beam\_search.py at c1f85598eb637bb7b21f8c3b15886aaed40ca42e · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/c1f85598eb637bb7b21f8c3b15886aaed40ca42e/src/transformers/generation/beam_search.py#L291) will just mark itself as finished. I don’t quite understand why an error is thrown here.

To replicate the issue:

```auto
import torch
import transformers
device='cuda:0'
input_ids = torch.tensor([[2, 33683, 209, 1142, 35, 50118, 1864, 35, 96, 3430, 10, 258, 219, 73, 17143, 324, 16, 10, 116, 50118, 250, 35, 446, 50118, 1864, 35, 3394, 56, 10, 1510, 29, 440, 112, 478, 19, 22426, 370, 404, 2306, 116, 50118, 250, 35]])
model = transformers.OPTForCausalLM.from_pretrained("facebook/opt-6.7b", torch_dtype=torch.float16)
model.to(device)
generation_config = {"bad_words_ids": [[2, 1864, 35] ], "early_stopping": True, "eos_token_id": [50118, 6, 4], "max_new_tokens": 256, "pad_token_id": 2}
generation_config = transformers.GenerationConfig(**generation_config)
model.generate(input_ids.to(device), attention_mask=torch.ones_like(input_ids).to(device),
               num_beams=5, num_return_sequences=1, 
               do_sample=False, generation_config=generation_config)

```

I noticed, however, if we don’t pass `attention_mask`, no error will be raised for this input. The behavior is reversed with the following input (i.e. error without `attention_mask` and no error with)

```auto
input_ids = torch.tensor([[2, 33683, 209, 1142, 35, 50118, 1864, 35, 96, 3430, 10, 258, 219, 73, 17143, 324, 16, 10, 116, 50118, 250, 35, 446, 50118, 1864, 35, 3394, 21, 5, 313, 639, 20, 11055, 20614, 2258, 116, 50118, 250, 35]])

```

---

<div class="post-metadata">

**Author:** ![WaveCut](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/wavecut/32/43465_2.png) [@WaveCut](https://discuss.huggingface.co/u/WaveCut)\
**Post date:** [October 19, 2023, 12:00pm UTC](https://discuss.huggingface.co/t/potential-bug-with-beam-search-eos-token-id/35862/2 "2023-10-19T12:00:30Z")

</div>

Hey, I have the same issue, with the same inputs, with only difference is mode == BEAM\_SAMPLE. Had you luck mitigation it?  
@zl7