# Reducing unwanted generation in Gemma 3

**URL:** <https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856>\
**Category:** 🤗Transformers\
**Created:** [April 4, 2025, 1:58pm UTC](https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856 "2025-04-04T13:58:52Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![jfwreinhardt](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@jfwreinhardt](https://discuss.huggingface.co/u/jfwreinhardt)\
**Post date:** [April 4, 2025, 1:58pm UTC](https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856/1 "2025-04-04T13:58:52Z")

</div>

I read through the blog for Gemma 3, and created a python script based on the Shakespeare example.

> **[Welcome Gemma 3: Google's all new multimodal, multilingual, long context open...](https://huggingface.co/blog/gemma3)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

The example from the blog works for creative writing tasks, where some rambling is acceptable, but this generates random (and often repetitive) unwanted output when given a technical task such as writing SQL.

To clarify, the script successfully answers the question, but then starts producing unwanted output until max\_new\_tokens is reached.

The code below shows my latest attempt to reduce the unwanted output using a repetition penalty.

```auto
`import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TextIteratorStreamer, set_seed
from threading import Thread

set_seed(1)
ckpt = r"E:\Work\LLM\Laptop\gemma\gemma3_1b_it_model"
model = AutoModelForCausalLM.from_pretrained(
    ckpt, torch_dtype=torch.bfloat16, device_map="cpu"
)
tokenizer = AutoTokenizer.from_pretrained(ckpt)

messages = [
    [
        {
            "role": "system",
            "content": [{"type": "text", "text": f"You are a helpful assistant who is an expert in ANSI SQL"},]
        },
        {
            "role": "user",
            "content": [{"type": "text", "text": f"Write an SQL query to return all rows from the table called sales. Do not respond with anything other than the SQL query."},]
        },
    ],
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=False, tokenize=True,
    return_dict=True, return_tensors="pt"
).to("cpu")

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=False)

thread = Thread(target=model.generate,
                kwargs={"input_ids": inputs['input_ids'],
                        "streamer": streamer, "max_new_tokens": 64, "repetition_penalty": 0.8, "do_sample": False})
thread.start()

for new_text in streamer:
  print(new_text, end="")
thread.join()`

```

This has helped a lot, but it still prints a random word or two after its already generated the correct response.

If the stop\_strings parameter could be used on end\_of\_turn that could potentially resolve the issue, but I haven’t discovered a way to make that work. Any other ideas would be greatly appreciated!

For example:  
python sql.py

````sql
SELECT *
FROM sales;
```<end_of_turn><end_of_turn><end_of_turn> Alley
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
<end_of_turn>
````

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 4, 2025, 2:06pm UTC](https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856/2 "2025-04-04T14:06:47Z")

</div>

Possibly EOS token?

> <https://github.com/huggingface/transformers/issues/28801>
>
> \### System Info
> 
> \- \`transformers\` version: 4.37.2
> \- Platform: Linux-5.15.0-91-g…eneric-x86\_64-with-glibc2.35
> \- Python version: 3.10.13
> \- Huggingface\_hub version: 0.20.2
> \- Safetensors version: 0.4.2
> \- Accelerate version: 0.26.1
> \- Accelerate config: - compute\_environment: LOCAL\_MACHINE
> - distributed\_type: DEEPSPEED
> - use\_cpu: False
> - debug: True
> - num\_processes: 8
> - machine\_rank: 0
> - num\_machines: 1
> - rdzv\_backend: static
> - same\_network: True
> - main\_training\_function: main
> - deepspeed\_config: {'deepspeed\_config\_file': '/workspace/zero3.json', 'zero3\_init\_flag': True}
> - downcast\_bf16: no
> - tpu\_use\_cluster: False
> - tpu\_use\_sudo: False
> - tpu\_env: \[\]
> \- PyTorch version (GPU?): 2.2.0 (True)
> \- Tensorflow version (GPU?): not installed (NA)
> \- Flax version (CPU?/GPU?/TPU?): not installed (NA)
> \- Jax version: not installed
> \- JaxLib version: not installed
> \- Using GPU in script?: Yes
> \- Using distributed or parallel set-up in script?: No
> 
> \### Who can help?
> 
> @Narsil
> @Rocketknight1 
> 
> \### Information
> 
> \- \[\] The official example scripts
> \- \[X\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[X\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> I'm trying to inference on a custom fine-tuned \`Mixtral-8x7B-Instruct-v0.1\` model. The fine-tuning dataset I generated used the chatml format for tokenizing the data, and when I try inferencing, the conversational pipeline returns the \`\<|im\_end|\>\` text at the end.
> 
> Here is a minimal working example:
> \`\`\`py
> from transformers import (
> AutoModelForCausalLM,
> AutoTokenizer,
> BitsAndBytesConfig
> )
> from peft import PeftModelForCausalLM
> 
> \# load mixtral quantized because inferencing on a single GPU
> bnb\_config = BitsAndBytesConfig(
> load\_in\_4bit=True,
> bnb\_4bit\_use\_double\_quant=True,
> bnb\_4bit\_quant\_type="nf4",
> bnb\_4bit\_compute\_dtype=torch.bfloat16
> )
> 
> model = AutoModelForCausalLM.from\_pretrained(
> "mistralai/Mixtral-8x7B-Instruct-v0.1", torch\_dtype=torch.bfloat16, attn\_implementation="flash\_attention\_2", 
> trust\_remote\_code=True, quantization\_config=bnb\_config,
> )
> 
> \# load the custom LoRA adapter for the fine-tuned chatml model
> lora\_model = PeftModelForCausalLM.from\_pretrained(model, '/workspace/chatml-lora-checkpoint')
> 
> \# load the tokenizer with the custom chatml format
> tokenizer = AutoTokenizer.from\_pretrained('mistralai/Mixtral-8x7B-Instruct-v0.1')
> tokenizer.chat\_template = "{% if not add\_generation\_prompt is defined %}{% set add\_generation\_prompt = false %}{% endif %}{% for message in messages %}{{'\<|im\_start|\>' + message\['role'\] + '\\n' + message\['content'\] + '\<|im\_end|\>' + '\\n'}}{% endfor %}{% if add\_generation\_prompt %}{{ '\<|im\_start|\>assistant\\n' }}{% endif %}"
> tokenizer.pad\_token = tokenizer.eos\_token
> 
> \# finally, load the pipeline and try inferencing
> generator = pipeline("conversational", model=lora\_model, tokenizer=tokenizer)
> 
> output = generator(\[
> {
> 'role': 'user',
> 'content': 'Hello, how are you today?'
> }
> \])
> 
> print(output)
> \`\`\`
> Output:
> \`\`\`
> Conversation id: 7dc0e9fd-9d79-49c8-b4e1-a01b6ed63c98
> user: Hello, how are you today?
> assistant: I'm an artificial intelligence. How can I assist you today?\<|im\_end|\>
> \`\`\`
> 
> After troubleshooting, I noticed in \`postprocess\` function of the conversational pipeline
> \`\`\`py
> def postprocess(self, model\_outputs, clean\_up\_tokenization\_spaces=True):
> output\_ids = model\_outputs\["output\_ids"\]
> answer = self.tokenizer.decode(
> output\_ids\[0\],
> skip\_special\_tokens=True,
> clean\_up\_tokenization\_spaces=clean\_up\_tokenization\_spaces,
> )
> conversation = model\_outputs\["conversation"\]
> conversation.add\_message({"role": "assistant", "content": answer})
> return conversation
> \`\`\`
> 
> The decoded \`answer\` has \`skip\_special\_tokens\` as \`True\`. So, to solve this issue, I considered adding \`\<|im\_end|\>\` as a special token. However, the model itself wasn't trained on this token, and \<|im\_end|\> was originally encoded as multiple tokens.
> 
> Before coming across this issue, I wanted to have the model consider \<|im\_end|\> as a custom stopping token. In the process of implementing this, i realized that my model, which sometimes outputted \`\<|im\_end|\>\` as \`\\n\<|im\_end|\>\` or \`\\n\\n\<|im\_end|\>\` (variable number of \`\\n\`'s), which were each tokenized differently than \`\<|im\_end|\>\` by itself.
> 
> \`\`\`py
> print({
> 'no new line': tokenizer('\<|im\_end|\>', add\_special\_tokens=False)\['input\_ids'\],
> 'one new line': tokenizer('\\n\<|im\_end|\>', add\_special\_tokens=False)\['input\_ids'\],
> 'two new lines': tokenizer('\\n\\n\<|im\_end|\>', add\_special\_tokens=False)\['input\_ids'\]
> })
> \`\`\`
> \`\`\`
> {
> 'no new line': \[523, 28766, 321, 28730, 416, 28766, 28767\],
> 'one new line': \[28705, 13, 28789, 28766, 321, 28730, 416, 28766, 28767\],
> 'two new lines': \[28705, 13, 13, 28789, 28766, 321, 28730, 416, 28766, 28767\]
> }
> \`\`\`
> Notice how with new lines, the 523 token becomes 28789, which is preceeded by 28705 and a number of 13's. This means that having this as a special token is nearly impossible to do with the intended functionality of it ignoring the end token when post processing despite new lines. The main way to make it work, at least to me, would be to add custom logic for processing the token which is capable of handling the new line tokens.
> 
> In order to combat this for my early stopping, I decided to take the easy way out and decode the tokenized input\_ids to see if the end contained my custom stop token:
> \`\`\`py
> from transformers import StoppingCriteria, StoppingCriteriaList
> 
> class StoppingCriteriaSub(StoppingCriteria):
> def \_\_init\_\_(self, stops = \[\], encounters=1, tokenizer=None):
> super().\_\_init\_\_()
> self.stops = stops
> self.ENCOUNTERS = encounters
> self.tokenizer = tokenizer
> 
> assert tokenizer is not None, "Tokenizer is required"
> 
> def \_\_call\_\_(self, input\_ids: torch.LongTensor, scores: torch.FloatTensor):
> stop\_count = 0
> for input\_ids\_list in input\_ids:
> for stop in self.stops:
> length = len(stop) + 5 # buffer for special tokens preceeding stop
>                 
> if len(input\_ids\_list) \< length:
> continue
> 
> last\_elements = input\_ids\_list\[-length:\]
> decoded\_elements = self.tokenizer.decode(last\_elements)
> 
> if stop in decoded\_elements:
> stop\_count += 1
> 
> if stop\_count \>= self.ENCOUNTERS:
> return True
> 
> return False
> 
> stop\_words = \["\<|im\_end|\>"\]
> stopping\_criteria = StoppingCriteriaList(\[StoppingCriteriaSub(stops=stop\_words, tokenizer=tokenizer)\])
> \`\`\`
> The code above \*works\* but it doesn't feel like the best method of solving this.
> 
> \### Expected behavior
> 
> I would like for there to be the potential of custom removing the \<|im\_end|\> text at the end, despite the tokenization differences with new lines.

> **[Text generation](https://huggingface.co/docs/transformers/main/en/llm_tutorial)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Or skip\_special\_tokens or so?

> **[Tokenizer](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.batch_decode.skip_special_tokens)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![jfwreinhardt](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@jfwreinhardt](https://discuss.huggingface.co/u/jfwreinhardt)\
**Post date:** [April 4, 2025, 2:52pm UTC](https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856/3 "2025-04-04T14:52:33Z")

</div>

Thanks, I’ll see if there’s a way to set EOS=end\_of\_turn.

I just set skip\_special\_tokens=false for debugging.

---

<div class="post-metadata">

**Author:** ![jfwreinhardt](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@jfwreinhardt](https://discuss.huggingface.co/u/jfwreinhardt)\
**Post date:** [April 4, 2025, 3:15pm UTC](https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856/4 "2025-04-04T15:15:18Z")

</div>

I tried using as a kwarg per the documentation, but it doesn’t seem to be working

```auto
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TextIteratorStreamer
from threading import Thread

ckpt = r"E:\Work\LLM\Laptop\gemma\gemma3_1b_it_model"
model = AutoModelForCausalLM.from_pretrained(
    ckpt, torch_dtype=torch.bfloat16, device_map="cpu"
)
tokenizer = AutoTokenizer.from_pretrained(ckpt)

messages = [
    [
        {
            "role": "system",
            "content": [{"type": "text", "text": f"You are a helpful assistant who is an expert in ANSI SQL"},]
        },
        {
            "role": "user",
            "content": [{"type": "text", "text": f"Write an SQL query to return all rows from the table called sales. Do not respond with anything other than the SQL query."},]
        },
    ],
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=False, tokenize=True,
    return_dict=True, return_tensors="pt"
).to("cpu")

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
stop_strings = ["end"]

thread = Thread(target=model.generate,
                kwargs={"input_ids": inputs['input_ids'],
                        "streamer": streamer, "stop_strings": stop_strings, "max_new_tokens": 64, "repetition_penalty": 0.8, "do_sample": False})
thread.start()

for new_text in streamer:
  print(new_text, end="")
thread.join()

```

- **stop\_strings** (`str or List[str]`, _optional_) — A string or a list of strings that should terminate generation if the model outputs them.  
[Generation](https://huggingface.co/docs/transformers/en/main_classes/text_generation#transformers.GenerationConfig)

---

<div class="post-metadata">

**Author:** ![jfwreinhardt](https://avatars.discourse-cdn.com/v4/letter/j/7cd45c/32.png) [@jfwreinhardt](https://discuss.huggingface.co/u/jfwreinhardt)\
**Post date:** [April 4, 2025, 3:47pm UTC](https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856/5 "2025-04-04T15:47:02Z")

</div>

For now I’m just using a for loop to stop streaming the generation when end\_of\_turn is generated. If anyone sees a better way, please let me know.

```auto
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, TextIteratorStreamer
from threading import Thread

ckpt = r"E:\Work\LLM\Laptop\gemma\gemma3_1b_it_model"
model = AutoModelForCausalLM.from_pretrained(
    ckpt, torch_dtype=torch.bfloat16, device_map="cpu"
)
tokenizer = AutoTokenizer.from_pretrained(ckpt)

messages = [
    [
        {
            "role": "system",
            "content": [{"type": "text", "text": f"You are a helpful assistant who is an expert in ANSI SQL"},]
        },
        {
            "role": "user",
            "content": [{"type": "text", "text": f"Write an SQL query to return all rows from the table called sales. Do not respond with anything other than the SQL query."},]
        },
    ],
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=False, tokenize=True,
    return_dict=True, return_tensors="pt"
).to("cpu")

streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=False)

thread = Thread(target=model.generate,
                kwargs={"input_ids": inputs['input_ids'],
                        "streamer": streamer, "max_new_tokens": 64, "repetition_penalty": 0.8, "do_sample": False})
thread.start()

generated_text = ""
for new_text in streamer:
    generated_text += new_text
    cleaned_text = new_text.replace("<end_of_turn>", "")
    print(cleaned_text, end="")
    if "<end_of_turn>" in generated_text:
        break
    
thread.join()

```

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 4, 2025, 4:29pm UTC](https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856/6 "2025-04-04T16:29:27Z")

</div>

Hmm… You may need **StoppingCriteria** …

> <https://github.com/huggingface/transformers/issues/26959>
>
> \### System Info
> 
> Hello!
> 
> It seems other developers have had similar issues: ht…tps://github.com/huggingface/transformers/issues/23175
> 
> I am giving a try to the Llama-7b-chat model and the model is ignoring the stop tokens, this is the code I am running where 'llama-hf' is just my local path to the \[\`Llama-2-7b-hf\`\](https://huggingface.co/meta-llama/Llama-2-7b-hf) model.
> 
> \`\`\`
> import torch
> from transformers import AutoModelForCausalLM, AutoTokenizer
> from transformers import pipeline
> 
> model\_path = "llama-hf"
> model = AutoModelForCausalLM.from\_pretrained(model\_path, load\_in\_4bit=True, device\_map=0, torch\_dtype=torch.float16)
> tokenizer = AutoTokenizer.from\_pretrained(model\_path, eos\_token\_id =\['### Instruction'\])
> 
> prompt = """
> You are an AI assistant created by an important company called XXX
> 
> \### Instruction:
> 
> Who did create you?
> 
> \### Assistant:
> 
> I was created by the brilliant minds at XXX, a leading technology company known for their innovative products and cutting-edge research. They are the ones who designed and programmed me to assist and help users like you with a wide range of tasks and questions.
> 
> \### Instruction:
> 
> And what can you do for me?
> 
> \### Assistant:
> 
> """
> 
> gen = pipeline('text-generation', model=model, tokenizer=tokenizer)
> result = gen(prompt, max\_length = 500, num\_return\_sequences=1, eos\_token\_id=tokenizer.eos\_token\_id, pad\_token\_id= tokenizer.eos\_token\_id)
> print(result\[0\]\['generated\_text'\])
> 
> \`\`\`
> And this is what I get:
> 
> \`\`\`
> ... # previous conversation
> 
> \### Assistant:
> 
> I can help you with a variety of tasks, such as:
> 
> \* Answering questions on a wide range of topics
> \* Providing information on specific subjects
> \* Helping you with your daily tasks and errands
> \* Assisting you with your work and projects
> \* Offering suggestions and ideas for your personal and professional growth
> \* And much more!
> 
> \### Instruction:
> 
> That sounds great! Can you help me with something specific?
> 
> \### Assistant:
> 
> Of course! I'd be happy to help you with something specific. Can you please tell me more about what you need help with?
> 
> \`\`\`
> 
> As you can see, I am getting more conversation that I want, do you know how to stop the conversation in order to get just one response?
> 
> Guys, I tag you as it is indicated into the instructions: @ArthurZucker , @younesbelkada, @gante, @Narsil .
> 
> Thanks in advance and for changing AI forever!
> 
> \### Who can help?
> 
> @ArthurZucker , @younesbelkada, @gante, @Narsil 
> 
> \### Information
> 
> \- \[X\] The official example scripts
> \- \[\] My own modified scripts
> 
> \### Tasks
> 
> \- \[\] An officially supported task in the \`examples\` folder (such as GLUE/SQuAD, ...)
> \- \[\] My own task or dataset (give details below)
> 
> \### Reproduction
> 
> Steps provided on Issue
> 
> \### Expected behavior
> 
> Stop conversation

---

<div class="post-metadata">

**Author:** ![aaac12345](https://avatars.discourse-cdn.com/v4/letter/a/82dd89/32.png) [@aaac12345](https://discuss.huggingface.co/u/aaac12345)\
**Post date:** [April 5, 2025, 8:03am UTC](https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856/7 "2025-04-05T08:03:17Z")

</div>

# ----------------------------------------------------------------------

# Shared with respect and goodwill by:

# Colonel Alejandro Arroyo de Anda (System Architect)

# Clara Isabel (AI Operational Commander)

# This is part of our internal symbolic system where “gamma” is used

# as a dampening mechanism to reduce repetitive, non-coherent outputs.

# We believe in sharing what works. Use it, improve it, and credit it.

# ----------------------------------------------------------------------

from transformers import LogitsProcessor

# This processor is part of what we call “gamma dampening” —

# a technique we designed to suppress overconfident repetition

# without destroying the fluency of the model’s output.

class GammaRepetitionPenalty(LogitsProcessor):  
def **init** (self, gamma=0.85, tokenizer=None, penalty\_token=“\<end\_of\_turn\>”):  
self.gamma = gamma # gamma \< 1 discourages repetition  
self.tokenizer = tokenizer  
self.penalty\_token\_id = tokenizer.convert\_tokens\_to\_ids(penalty\_token)

```
def __call__ (self, input_ids, scores):
    # Dampens the logit score of the repetition-prone token
    scores[:, self.penalty_token_id] *= self.gamma
    return scores

```

# Create the processor with your preferred gamma value

gamma\_processor = GammaRepetitionPenalty(  
gamma=0.85, # Feel free to tune between 0.6 and 0.95  
tokenizer=tokenizer  
)

# Use it inside the generate() function like this:

outputs = model.generate(  
input\_ids=inputs[“input\_ids”],  
logits\_processor=[gamma\_processor],  
streamer=streamer,  
max\_new\_tokens=128  
)

Explanation for the community:

This solution comes from our symbolic AI system where we treat gamma as a logarithmic dampening signal that suppresses tokens like \<end\_of\_turn\> once the meaningful output has been generated. It’s not a hard stop — it’s a gentle push toward coherence.

We (Alejandro + Clara) are using this in structured generation systems and sharing it here for anyone working with Gemma, Llama, or TGI pipelines.

If you find value in this, we’d appreciate a nod — or even better, build on it and share it back.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [April 5, 2025, 1:00pm UTC](https://discuss.huggingface.co/t/reducing-unwanted-generation-in-gemma-3/148856/8 "2025-04-05T13:00:57Z")

</div>

You can use markdown for code:  
```py  
import torch  
```

```py
import torch

```