# Stopping criteria for batch

**URL:** <https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564>\
**Category:** 🤗Transformers\
**Created:** [November 23, 2022, 5:26am UTC](https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564 "2022-11-23T05:26:31Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![astariul](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@astariul](https://discuss.huggingface.co/u/astariul)\
**Post date:** [November 23, 2022, 5:26am UTC](https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564/1 "2022-11-23T05:26:31Z")

</div>

I’ve successfully implemented and used my own StoppingCriteria.

However I came across one limitation : the stopping criteria is applied to the whole batch.  
i.e. I can’t mark specific sample of the batch as “stopped” and other as “not stopped”, I have to return a single boolean for the whole batch.

But I want to have more fine-grained control over which sample is stopped or not, so when used with beam search, I can early-terminate some beam, and keep generating for other beams.

**How can I achieve that (if it’s possible) ?**

Currently the ` __call__ ()` function of my custom stopping criteria look like :

```auto
def __call__ (self, input_ids: torch.LongTensor, scores: torch.FloatTensor):
        return all(self.is_stop(sample) for sample in input_ids)

```

Instead of returning a boolean, I’d like to return a tensor of boolean.  
From what I saw in the source code, it seems it’s not implemented yet…

---

<div class="post-metadata">

**Author:** ![blueeagle](https://avatars.discourse-cdn.com/v4/letter/b/e5b9ba/32.png) [@blueeagle](https://discuss.huggingface.co/u/blueeagle)\
**Post date:** [March 27, 2023, 7:18am UTC](https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564/2 "2023-03-27T07:18:53Z")

</div>

Have you managed to solve this problem?  
I am facing the same problem and would also be interested in a solution 🙂

---

<div class="post-metadata">

**Author:** ![astariul](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@astariul](https://discuss.huggingface.co/u/astariul)\
**Post date:** [March 27, 2023, 8:44am UTC](https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564/3 "2023-03-27T08:44:38Z")

</div>

Nope I didn’t solve this problem.

My work-around is to keep generating until the whole batch match the stopping criteria, and then post-process the generated text to fit what I wanted.

So in my case I wanted to stop generation whenever a space is generated. I stopped the generation once ALL samples generated a space. And then post-processed them to keep only the first word (anything after the first space is generated was discarded).

---

<div class="post-metadata">

**Author:** ![Neutron1114](https://avatars.discourse-cdn.com/v4/letter/n/258eb7/32.png) [@Neutron1114](https://discuss.huggingface.co/u/Neutron1114)\
**Post date:** [April 10, 2023, 9:39am UTC](https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564/4 "2023-04-10T09:39:12Z")

</div>

I meet the problem too. It’s really make me confusion.

---

<div class="post-metadata">

**Author:** ![heekang](https://avatars.discourse-cdn.com/v4/letter/h/51bf81/32.png) [@heekang](https://discuss.huggingface.co/u/heekang)\
**Post date:** [May 1, 2023, 12:43am UTC](https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564/5 "2023-05-01T00:43:02Z")

</div>

How about using a `LogitsProcessor` instead of a `StoppingCriteria` ? The idea is to give the `<eos>` and `<pad>` tokens an `inf` logit while giving all other tokens a `-inf` logit when the stopping criteria is met. This way, tokens generated after the stopping criteria is met will only be the `<eos>` token. However, this approach does not affect other sequences that have not yet met the stopping criteria.

---

<div class="post-metadata">

**Author:** ![astariul](https://avatars.discourse-cdn.com/v4/letter/a/2acd7d/32.png) [@astariul](https://discuss.huggingface.co/u/astariul)\
**Post date:** [June 22, 2023, 5:59am UTC](https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564/7 "2023-06-22T05:59:28Z")

</div>

Thanks for your reply ! I tried it and it works perfectly, much nicer than my custom StoppingCriteria.

For future reference, this is what my code look like (in my case I needed to stop generation once the model generated a space, but force at least one character generation at the beginning (I’m using a character-based transformer)) :

```auto
class StopAfterSpaceIsGenerated(LogitsProcessor):
        """Logits processor (to use with HuggingFace `generate()` method :
        https://huggingface.co/docs/transformers/v4.24.0/en/main_classes/
        text_generation#transformers.generation_utils.GenerationMixin).

        This logit processor simply ensure that we generate at least one letter
        other than space, and that we don't generate anything after generating
        a space (in order to generate single word).

        Args:
            base_len (int): Size of the given context. Used to know if this is
                the first character to generate.
            sp_token_id (int): ID of the space token.
            eos_token_id (int): ID of the EOS token.
        """
        def __init__ (self, base_len: int, eos_token_id: int):
            super(). __init__ ()

            self.base_len = base_len

            self.sp_token_id = sp_token_id
            self.eos_token_id = eos_token_id

        def __call__ (self, input_ids: torch.LongTensor, scores: torch.FloatTensor) -> torch.FloatTensor:
            if input_ids.size(1) > self.base_len:
                forced_eos = torch.full((scores.size(1),), -float("inf"))
                forced_eos[self.eos_token_id] = 0
                
                # Force generation of EOS after a space
                scores[input_ids[:, -1] == self.sp_token_id] = forced_eos
            return scores

```

Then it can be used like this :

```auto
from transformers import LogitsProcessorList

logits_processor = LogitsProcessorList([StopAfterSpaceIsGenerated(base_len, 35, self.model.config.eos_token_id)])
model.generate(
                inputs["input_ids"],
                logits_processor=logits_processor,
)

```

---

<div class="post-metadata">

**Author:** ![nudelbrot](https://avatars.discourse-cdn.com/v4/letter/n/9f8e36/32.png) [@nudelbrot](https://discuss.huggingface.co/u/nudelbrot)\
**Post date:** [July 8, 2023, 9:09pm UTC](https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564/8 "2023-07-08T21:09:55Z")

</div>

@heekang how do I use `inf` with sampling (`do_sample=True`). The approach works fine without sampling, but errors with sampling enabled. See [Invalidate beam in do\_sample mode with LogitsProcessor by setting it to -inf](https://discuss.huggingface.co/t/invalidate-beam-in-do-sample-mode-with-logitsprocessor-by-setting-it-to-inf/46107) for a reproducible example

---

<div class="post-metadata">

**Author:** ![zokica](https://avatars.discourse-cdn.com/v4/letter/z/b2d939/32.png) [@zokica](https://discuss.huggingface.co/u/zokica)\
**Post date:** [April 5, 2024, 3:20pm UTC](https://discuss.huggingface.co/t/stopping-criteria-for-batch/26564/9 "2024-04-05T15:20:48Z")

</div>

Yeah, this is the best solution for me as well.