# How can I specify \`stop\_strings\` in \`generation\_config.json\`?

**URL:** <https://discuss.huggingface.co/t/how-can-i-specify-stop-strings-in-generation-config-json/114611>\
**Category:** 🤗Transformers\
**Created:** [October 29, 2024, 11:19pm UTC](https://discuss.huggingface.co/t/how-can-i-specify-stop-strings-in-generation-config-json/114611 "2024-10-29T23:19:06Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![suhara](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@suhara](https://discuss.huggingface.co/u/suhara)\
**Post date:** [October 29, 2024, 11:19pm UTC](https://discuss.huggingface.co/t/how-can-i-specify-stop-strings-in-generation-config-json/114611/1 "2024-10-29T23:19:07Z")

</div>

I have a model with which I want to use `stop_strings` to terminate generation with certain keywords.

I know `stop_strings` has to be accompanied with a tokenizer object like below.

```auto
model.generate(...,
               stop_strings=["<stop token>"],
               tokenizer=tokenizer)

```

I’m wondering if I can add this setting to any \*config.json.

When I add `stop_strings` to `generation_config.json` and try to generate

```auto
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained(model_path)
>>> model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")
>>> model.generate(**tokenizer("Hi how are you?", return_tensors="pt", return_token_type_ids=False))

...

ValueError: There are one or more stop strings, either in the arguments to `generate` or in the model's generation config, but we could not locate a tokenizer. When generating with stop strings, you must pass the model's tokenizer to the `tokenizer` argument of `generate`.

```

Is there a way to define `stop_strings` and the default tokenizer so I can skip manually passing the info whenever calling `model.generate()`?

---

<div class="post-metadata">

**Author:** ![suhara](https://avatars.discourse-cdn.com/v4/letter/s/71c47a/32.png) [@suhara](https://discuss.huggingface.co/u/suhara)\
**Post date:** [November 14, 2024, 6:01pm UTC](https://discuss.huggingface.co/t/how-can-i-specify-stop-strings-in-generation-config-json/114611/2 "2024-11-14T18:01:14Z")

</div>

Apparently, it’s not supported yet.

> <https://github.com/huggingface/transformers/issues/34571>
>
> \### Feature request
> 
> The transformer library should offer a way to configure \`st…op\_strings\` and the tokenizer for it.
> 
> \`model.generate()\` can take a \`stop\_strings\` argument to use custom stop tokens for generation, but a tokenizer object needs to be passed as well.
> 
> \`\`\`
> model.generate(...,
> stop\_strings=\["\<stop token\>"\],
> tokenizer=tokenizer)
> \`\`\`
> 
> If we add \`stop\_strings\` to \`generation\_config.json\`, which can be loaded correctly \[code\](https://github.com/huggingface/transformers/blob/33868a057c02f0368ba63bd1edb746be38fe3d90/src/transformers/generation/configuration\_utils.py#L144-L145), it will return the following error, as it requires a tokenizer object, which cannot be defined in the config file. 
> 
> \`\`\`
> \>\>\> from transformers import AutoTokenizer, AutoModelForCausalLM
> \>\>\> tokenizer = AutoTokenizer.from\_pretrained(model\_path)
> \>\>\> model = AutoModelForCausalLM.from\_pretrained(model\_path, device\_map="auto")
> \>\>\> model.generate(\*\*tokenizer("Hi how are you?", return\_tensors="pt", return\_token\_type\_ids=False))
> 
> ...
> 
> ValueError: There are one or more stop strings, either in the arguments to \`generate\` or in the model's generation config, but we could not locate a tokenizer. When generating with stop strings, you must pass the model's tokenizer to the \`tokenizer\` argument of \`generate\`.
> \`\`\`
> 
> 
> \### Motivation
> 
> The user shouldn't be bothered by adding extra arguments to \`generate()\` or \`pipeline\`.
> 
> For example, \[nvidia/Mistral-NeMo-Minitron-8B-Instruct\](https://huggingface.co/nvidia/Mistral-NeMo-Minitron-8B-Instruct) needs to use \`stop\_strings\` but so many people simply calls \`generate()\` without \`stop\_strings\` and share complaints.
> 
> 
> \### Your contribution
> 
> I'd be happy to create a PR but need guidance for the design choice.