# How does the text-generation pipeline know the special stop token? **URL:** \ **Category:** Beginners\ **Created:** [June 5, 2024, 7:01am UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412 "2024-06-05T07:01:14Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![VitalContribution](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vitalcontribution/32/27815_2.png) [@VitalContribution](https://discuss.huggingface.co/u/VitalContribution)\ **Post date:** [June 5, 2024, 7:01am UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412/1 "2024-06-05T07:01:14Z")
I use the Llama2 model currently which has the stop token ``. I tried to change the stop token so that the pipeline would continue to generate regardless of the model predicting the `` token. I tried to change: ```python model.config.eos_token_id = # or tokenizer.eos_token = "somethingelse" ``` but the pipeline still seems to know that it should stop at `` So I’m asking myself: “How does the the Pipeline (Text-Generation) know what the correct stop token is and does it get the stop token from?” ---
**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\ **Post date:** [June 5, 2024, 10:44am UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412/2 "2024-06-05T10:44:41Z")
Hey! You have to change it also in model’s generation config. Try out the below code ```auto model.generation_config.eos_token_id = None model.config.eos_token_id = None ``` ---
**Author:** ![VitalContribution](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vitalcontribution/32/27815_2.png) [@VitalContribution](https://discuss.huggingface.co/u/VitalContribution)\ **Post date:** [June 5, 2024, 6:26pm UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412/3 "2024-06-05T18:26:01Z")
Thank you your response! I had no idea about the generation config. The thing is actually my pipeline does not stop to generate. The following print statements print all `` (the correct stop token). ```python print("1. ", tokenizer.decode(tokenizer.eos_token_id)) print("2. ", tokenizer.decode(model.config.eos_token_id)) print("3. ", tokenizer.decode(model.generation_config.eos_token_id)) ``` However, when I create and call the pipeline: ```python pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=200) # Run text generation pipeline with our next model while True: user_input = "write an sql statement" prompt = f"[INTS]{user_input}[/INST]" result = pipe(prompt) print(result[0]['generated_text']) ``` It outputs: ```auto [INTS]write an sql statement[/INST] SELECT * FROM table_name WHERE name = "mario"\n\nWhat does the SQL statement mean?\n\nIn this SQL... ``` You can see that that the correct stop token is set in every single config and the pipeline still ignores the `` and continues. ---
**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\ **Post date:** [June 6, 2024, 4:57am UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412/4 "2024-06-06T04:57:55Z")
Ah sorry, I thought you want to generate even if there is stop token. Then you shouldn’t be setting them to `None` as I showed above. If the model is still generating beyond eos\_token set in the `model.generation_config.eos_token_id`, can you check if the generation config has `forced_eos_tokens` or `min_length`. Also it would be easier if you provide a reproducible code with the model checkpoint you’re using 🙂 ---
**Author:** ![VitalContribution](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vitalcontribution/32/27815_2.png) [@VitalContribution](https://discuss.huggingface.co/u/VitalContribution)\ **Post date:** [June 6, 2024, 6:30am UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412/5 "2024-06-06T06:30:03Z")
No, I’m sorry, this was on me! The script is about a fine tuned LLM (with LoRA). It tries to predict the SQL instruction from a user question and the database table. Here is the entire script: [Google Colab](https://colab.research.google.com/drive/1zjthCqpm8EA_Twoa17Dk3_2536x63d-3?usp=sharing) Let me give you a quick overview: 1. Loading the base model and merging it with my trained LORA Adapter. 2. Showing all model configs 3. Showing all eos token ids (decoded) 4. Trying out the pipeline with the config files 5. Trying out the pipeline and setting specifically the `` stop token (also had no effect) ---
**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\ **Post date:** [June 6, 2024, 12:40pm UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412/6 "2024-06-06T12:40:43Z")
Thanks a lot for sharing the notebook. Unfortunately I still can’t reproduce it on ‘main’. I am using an arbitrary adapter as I don’t have access to the one you had in Colab, so my wild guess is that there’s smth in the adapter config as the version in Colab is the latest one. Let me know if you can provide the whole script with the adapter, and I will try to find the bug! ---
**Author:** ![VitalContribution](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vitalcontribution/32/27815_2.png) [@VitalContribution](https://discuss.huggingface.co/u/VitalContribution)\ **Post date:** [June 6, 2024, 6:06pm UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412/7 "2024-06-06T18:06:29Z")
No, thank you so much for even trying to help me. But please don’t spend too much time on it but if you can not reproduce it with these script I will go crazy. My trained lora adapter: [SQLFormerLlama2\_Adapter - Google Drive](https://drive.google.com/drive/folders/1KfBWUg_uLMC8QNmEfBCsGdnJSO0YJQ0F?usp=sharing) The entire training script: [Google Colab](https://colab.research.google.com/drive/1JGJZx4YrhomD_b41RytRsSyINTMNOdXp?usp=sharing) (If you want to try out the training script you just have to let it run for maybe 3-5min and then you can stop the cell. By then the model will have learned the rough desired structure and will already fail to stop at the stop token) ---
**Author:** ![RaushanTurganbay](https://avatars.discourse-cdn.com/v4/letter/r/848f3c/32.png) [@RaushanTurganbay](https://discuss.huggingface.co/u/RaushanTurganbay)\ **Post date:** [June 10, 2024, 5:32am UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412/8 "2024-06-10T05:32:07Z")
@VitalContribution hey, I ran the notebook with your adapter and I found the error. The generated “” tokens are not EOS, i.e. the model is generating ids: [829, 29879, 29958]`which are together decoded as “”. I can’t say right now what went wrong with training, or how to train properly since I’m prioritizing other tasks. A workaround here will be to use a `stop_strings` as a kwarg to pipeline, which stops when a custom string is generated. See [docs](https://huggingface.co/docs/transformers/en/internal/generation_utils#transformers.StopStringCriteria) for more details 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [June 11, 2024, 11:03am UTC](https://discuss.huggingface.co/t/how-does-the-text-generation-pipeline-know-the-special-stop-token/90412/9 "2024-06-11T11:03:29Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.