# Num\_return\_sequences \> num\_beams

**URL:** <https://discuss.huggingface.co/t/num-return-sequences-num-beams/170391>\
**Category:** 🤗Transformers\
**Created:** [November 13, 2025, 12:19pm UTC](https://discuss.huggingface.co/t/num-return-sequences-num-beams/170391 "2025-11-13T12:19:16Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![rvashurin](https://avatars.discourse-cdn.com/v4/letter/r/b77776/32.png) [@rvashurin](https://discuss.huggingface.co/u/rvashurin)\
**Post date:** [November 13, 2025, 12:19pm UTC](https://discuss.huggingface.co/t/num-return-sequences-num-beams/170391/1 "2025-11-13T12:19:16Z")

</div>

What is the reason to restrict num\_return\_sequences to be less or equal to num\_beams? Suppose I request 10 sequences to be generated with num\_beams=5. Consider the case when we already have 4 finished sequences, and at some subsequent decoding iteration the best candidate continuation is EoS. Why can’t we add this sequence as the fifth generated sequence and continue decoding with the same beam width from the next best 5 continuations that we have at this step?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [November 13, 2025, 12:19pm UTC](https://discuss.huggingface.co/t/num-return-sequences-num-beams/170391/2 "2025-11-13T12:19:17Z")

</div>



---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [November 13, 2025, 12:20pm UTC](https://discuss.huggingface.co/t/num-return-sequences-num-beams/170391/3 "2025-11-13T12:20:28Z")

</div>



---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 13, 2025, 1:19pm UTC](https://discuss.huggingface.co/t/num-return-sequences-num-beams/170391/4 "2025-11-13T13:19:33Z")

</div>

It doesn’t seem like there’s fundamentally important background to LLM that led to this implementation.

* * *

The reason in Hugging Face Transformers is not “deep theory,” it is:

1. How their beam-search implementation and data structures are written, and
2. The semantic meaning they chose for `num_beams` and `num_return_sequences`.

Your proposed behavior (keep 5 beams but accumulate 10 finished sequences) is a _different_ beam-search variant than what Transformers actually implements.

Below is a detailed, step-by-step explanation, tied directly to the docs and source.

* * *

## 1. What `num_beams` and `num_return_sequences` mean in Transformers

In `model.generate(...)` with beam search (i.e. `num_beams > 1`, `do_sample=False`):

- `num_beams`  
= beam width = how many partial hypotheses (beams) are kept alive at each decoding step.

- `num_return_sequences`  
= how many _finished beams_ you want back at the end.

Hugging Face’s “How to generate text” tutorial explains it like this (for beam search):

> Set `num_return_sequences` to the number of highest scoring beams that should be returned. Make sure though that `num_return_sequences <= num_beams`. ([Hugging Face](https://huggingface.co/blog/how-to-generate "How to generate text: using different decoding methods for ..."))

So in HF’s design:

> Beam search keeps `num_beams` candidate sequences per input; at the end, `num_return_sequences` just selects how many of those `num_beams` to return.

It is _not_ “target number of total distinct sequences, independent of beam width.”

* * *

## 2. Where the hard restriction comes from in the code

Inside `generate`, once it decides you’re in beam mode, it literally checks and throws:

```python
elif is_beam_gen_mode:
    if num_return_sequences > num_beams:
        raise ValueError("`num_return_sequences` has to be smaller or equal to `num_beams`.")
    ...
    return self.beam_search(
        input_ids,
        beam_scorer,
        ...
    )

```

The exact string appears multiple times (plain beam search, group beam search, constrained beam search). ([GitHub](https://raw.githubusercontent.com/huggingface/transformers/v4.23.1/src/transformers/generation_utils.py "https://raw.githubusercontent.com/huggingface/tran..."))

So the _public_ constraint is hard-coded: you simply cannot call beam search with `num_return_sequences > num_beams`.

That check reflects how the internal beam search is structured, so the more interesting question is: _why did they structure it that way?_

* * *

## 3. How Hugging Face beam search is structured internally

### 3.1. BeamSearchScorer and BeamHypotheses

Beam search is orchestrated by:

- `BeamSearchScorer`
- `BeamHypotheses` (one per example in the batch)

A nice high-level explanation of these internals is in Manuel de Prada’s “unofficial documentation for HF generate”: ([mdp\_website](https://manueldeprada.com/blog/posts/documenting-hf-generate/ "Unofficial documentation for the HuggingFace generation ..."))

- For each example, `BeamSearchScorer` holds a `BeamHypotheses` object.

- `BeamHypotheses`:

You can see the same logic in replicated/derived code (identical in structure to HF’s original):

```python
class BeamHypotheses(object):
    def __init__ (self, num_beams, max_length, length_penalty, early_stopping):
        self.num_beams = num_beams
        self.beams = []
        self.worst_score = 1e9

    def add(self, hyp, sum_logprobs):
        score = sum_logprobs / (len(hyp) ** self.length_penalty)
        if len(self) < self.num_beams or score > self.worst_score:
            self.beams.append((score, hyp))
            if len(self) > self.num_beams:
                # drop the worst beam
                ...

```

([Thank you very much.](https://www.cnblogs.com/nickchen121/p/15499576.html "Beam Search快速理解及代码解析- B站"))

Key property:

> For each example, at _all times_, the n-best list of finished hypotheses contains **at most `num_beams`** sequences.

When decoding ends, `BeamSearchScorer.finalize(...)` sorts those stored beams and selects the top `num_beam_hyps_to_keep` among them (and `num_beam_hyps_to_keep` is wired to `num_return_sequences`). ([Gist](https://gist.github.com/fzyzcjy/fab4bf82c62f23b3432123c84f14a2c6 "Speed up HuggingFace beam search by 10x"))

So internally the contract is:

- “I will track `num_beams` ongoing beams per example.”
- “I will keep up to `num_beams` finished beams per example.”
- “At the end, I will return ≤ `num_beams` of those finished beams.”

That is the structural reason you cannot ask for more than `num_beams` outputs from beam search: **the algorithm literally never stores more than `num_beams` finished candidates.**

### 3.2. The stopping rule: first-come, first-served

On top of that, the _stopping condition_ that HF uses is a widely-used heuristic:

- Keep a set `F_t` of completed (EOS) sequences.
- Mark the example as done once `|F_t| == num_beams` (when `early_stopping=True`), or when completed sequences are good enough that unfinished beams cannot beat them. ([Hugging Face Forums](https://discuss.huggingface.co/t/implement-details-about-beam-search/34623 "Implement details about Beam Search - 🤗Transformers"))

Kasai et al. show that Hugging Face and fairseq both implement what they call **first-come, first-served (FCFS) beam decoding**:

> They keep a set of completed sequences and stop decoding when the size of this set reaches the beam size. ([arXiv](https://arxiv.org/abs/2204.05424 "A Call for Clarity in Beam Search: How It Works and When It Stops"))

Typical `is_done` logic inside `BeamHypotheses`:

```python
def is_done(self, best_sum_logprobs: float, cur_len: int) -> bool:
    if len(self) < self.num_beams:
        return False
    elif self.early_stopping:
        return True
    else:
        cur_score = best_sum_logprobs / cur_len**self.length_penalty
        return self.worst_score >= cur_score

```

([Hugging Face Forums](https://discuss.huggingface.co/t/implement-details-about-beam-search/34623 "Implement details about Beam Search - 🤗Transformers"))

So with default `early_stopping=True`, as soon as you have `num_beams` finished sequences, decoding stops for that example.

* * *

## 4. Now map your scenario to the actual implementation

Your thought experiment:

- Set `num_beams = 5`, ask for `num_return_sequences = 10`.

- Imagine we had 4 finished sequences already.

- At the next step, the best candidate continuation is EOS.

- You propose:

There are two levels of problems here: _front-end_ and _backend_.

### 4.1. Front-end: `generate` won’t even start

The high-level `generate` method will refuse this configuration:

```python
if num_return_sequences > num_beams:
    raise ValueError("`num_return_sequences` has to be smaller or equal to `num_beams`.")

```

([GitHub](https://raw.githubusercontent.com/huggingface/transformers/v4.23.1/src/transformers/generation_utils.py "https://raw.githubusercontent.com/huggingface/tran..."))

So you never reach your scenario in practice with stock HF beam search.

But assume that check were removed. What then?

### 4.2. Backend: you still never get more than 5 distinct outputs

Even without the check, the backend behavior would be:

1. While decoding, each time the top EOS candidate for that example is selected among the top 2×`num_beams` next candidates, `BeamSearchScorer.process(...)` calls `beam_hyp.add(...)` and _does not_ keep that EOS beam alive. ([mdp\_website](https://manueldeprada.com/blog/posts/documenting-hf-generate/ "Unofficial documentation for the HuggingFace generation ..."))

2. After you have added that 5th finished hypothesis:

3. Even if you set `early_stopping=False` so that decoding continues:

So structurally:

> With the current code, you can never get more than `num_beams` distinct final sequences out of beam search, regardless of how long you run.

Your proposed algorithm needs two changes that HF does _not_ implement:

1. Let the finished-hypotheses container hold more than `num_beams` sequences (e.g. cap by `num_return_sequences` instead).
2. Adjust the stopping rule so it doesn’t stop immediately when `len(F_t) == num_beams`.

Both changes are beyond a small tweak; they change the semantics of beam search.

* * *

## 5. Why did Transformers choose this design?

There are three main reasons: semantics, approximate top-K correctness, and simplicity.

### 5.1. Semantics: `num_beams` controls search, `num_return_sequences` is just a selector

Hugging Face chose a simple, user-friendly mental model:

- `num_beams` = how many beams to use in beam search.
- `num_return_sequences` = how many of those beams you want back at the end.

They codify that in docs and tutorials:

> “`num_return_sequences` is the number of highest scoring beams that should be returned. Make sure that `num_return_sequences <= num_beams`.” ([Hugging Face](https://huggingface.co/blog/how-to-generate "How to generate text: using different decoding methods for ..."))

So the restriction is not an afterthought; it is part of the API contract they present.

### 5.2. Approximate top-K logic: beam width bounds how many “good” hypotheses you can claim

Beam search is an approximate way to find top-K sequences under a scoring function (e.g. length-normalized log-probability).

Important consequence:

- If the beam width is B, the algorithm only ever keeps B partial paths at each step.
- Any candidate that _might_ end up as 6-th, 7-th, … 10-th best final sequence might have been pruned earlier because it was outside the top-B at some time step.

So:

> With `num_beams = 5`, the algorithm only has enough information to claim an approximate top-5 list. Anything beyond 5 is not a principled “6th best, 7th best, …” — it’s just “next best among whatever survived the pruning”.

By restricting `num_return_sequences <= num_beams`, Transformers avoids creating the illusion that it is giving you a meaningful “top-10” when the search width was only 5.

If you _really_ want 10 plausible high-scoring beams from beam search in HF, the supported pattern is:

```python
outputs = model.generate(
    **inputs,
    num_beams=10,
    num_return_sequences=10,
    do_sample=False, # pure beam search
)

```

### 5.3. Simplicity and resource bounds

By tying all these together to `num_beams`, HF gets a simple, predictable complexity:

- Active beams per example: `num_beams`.
- Finished hypotheses stored per example: ≤ `num_beams`.
- Outputs per example: ≤ `num_beams`.

Time and memory scale as O(`num_beams × vocab_size × max_length`) with no hidden multiplicative factor from storing many more finished sequences.

Your proposed variant (“keep 5 beams but store 10+ finished hypotheses”) is totally implementable, but it complicates:

- Memory (more finished sequences kept).
- Stopping logic (cannot just stop when |F\_t| == num\_beams).
- API semantics (now `num_return_sequences` partly controls search depth, not just output count).

Instead, HF keeps things simple and pushes “I want _many_ outputs” into sampling mode rather than changing beam semantics.

* * *

## 6. Is your proposed behavior _theoretically_ reasonable?

Yes. Conceptually:

- Maintain B active beams.
- Keep collecting finished sequences in a larger set F (say size up to K \> B).
- Only stop when |F| reaches K or some more principled optimality condition.

This is exactly the kind of change that Kasai et al. propose using a **patience factor `p`** :

- The standard HF/fairseq implementation is `p = 1` (stop when |F| = beam\_size).
- They generalize to `p > 1`, which lets you keep searching after you already have beam\_size finished sequences. ([arXiv](https://arxiv.org/abs/2204.05424 "A Call for Clarity in Beam Search: How It Works and When It Stops"))

With `beam_size = 5` and `p = 2`, you would conceptually allow up to 10 finished hypotheses. That is very close to what you describe.

The key point:

> This is theoretically and practically reasonable, but it is simply _not_ what HF’s current `generate()` implements.

To support it, HF would need to:

- Let `BeamHypotheses` store up to `num_beam_hyps_to_keep` (or `p * num_beams`), instead of `num_beams`.
- Change `is_done` to consider patience and/or a more optimal stopping rule (e.g. the one from Huang et al., “When to Finish? Optimal Beam Search”). ([arXiv](https://arxiv.org/abs/1809.00069 "When to Finish? Optimal Beam Search for Neural Text Generation (modulo beam size)"))

That is a more involved change than just removing the validation check.

* * *

## 7. How to get “10 candidates” in practice with HF

If you want many candidates and you do _not_ insist on beam-search top-K semantics, the standard HF pattern is to use sampling:

```python
outputs = model.generate(
    **inputs,
    do_sample=True,
    num_beams=1, # or low beam if you really want
    num_return_sequences=10, # many samples
    top_k=50, # or top_p, temperature, etc.
)

```

In sampling mode, HF generates multiple sequences by expanding the batch (`num_return_sequences`) rather than by increasing beam width, so the `num_return_sequences <= num_beams` constraint does _not_ need to apply conceptually. (There is/was even a GitHub issue complaining that the same check is applied in sampling where it’s not necessary.) ([GitHub](https://raw.githubusercontent.com/huggingface/transformers/v4.23.1/src/transformers/generation_utils.py "https://raw.githubusercontent.com/huggingface/tran..."))

So HF’s current design is:

- Beam search: “few, high-quality, approximately top-`num_beams` outputs” → `num_return_sequences <= num_beams`.
- Sampling: “many diverse outputs” → `num_return_sequences` can be large, independent of any `num_beams`.

* * *

## 8. Summary

Putting it all together:

- The restriction `num_return_sequences <= num_beams` exists because:

- Your scenario (“we already have 4 finished sequences, now we get a 5th EOS; why not keep decoding with 5 active beams until we have 10 finished sequences?”) describes a more general beam-search variant with a _patience factor_ and a larger finished-hypothesis capacity. That variant is sound and studied in the literature, but Transformers does not currently implement it.

- If you want more than `num_beams` outputs with HF today:

* * *

## 9. Suggested resources (with why they’re useful)

1. **HF blog – “How to generate text: using different decoding methods”**  
Explains greedy vs beam vs sampling, shows the intended meaning of `num_beams` and `num_return_sequences`, and explicitly states that `num_return_sequences <= num_beams` for beam search. ([Hugging Face](https://huggingface.co/blog/how-to-generate "How to generate text: using different decoding methods for ..."))

2. **HF docs – “Generation strategies”**  
Official explanation of generation modes and parameters (`num_beams`, `early_stopping`, etc.), useful to see the “official” semantics HF expects. ([Hugging Face](https://huggingface.co/docs/transformers/en/generation_strategies "Generation strategies"))

3. **Manuel de Prada – “Unofficial documentation for the HuggingFace generation internals”**  
Very clear walkthrough of `beam_search`, `BeamSearchScorer`, and `BeamHypotheses`. Shows concretely that `BeamHypotheses` keeps only the best `n_beams` hypotheses. ([mdp\_website](https://manueldeprada.com/blog/posts/documenting-hf-generate/ "Unofficial documentation for the HuggingFace generation ..."))

4. **Kasai et al., “A Call for Clarity in Beam Search: How It Works and When It Stops”**  
Analyzes HF and fairseq beam search, proves they use a first-come-first-served stopping rule, and introduces the patience factor that would let you do what you propose (more finished hypotheses than the beam size). ([arXiv](https://arxiv.org/abs/2204.05424 "A Call for Clarity in Beam Search: How It Works and When It Stops"))

5. **Huang et al., “When to Finish? Optimal Beam Search for Neural Text Generation”**  
Gives a more principled stopping criterion for beam search and contrasts it with heuristics like “stop when you have `k` completed sequences.” Helpful if you want to design a more theoretically grounded variant. ([arXiv](https://arxiv.org/abs/1809.00069 "When to Finish? Optimal Beam Search for Neural Text Generation (modulo beam size)"))

These will give you both the exact behavior of Transformers’ implementation and the broader context of how that behavior fits into the space of possible beam-search algorithms.