# SFT of Instruct Model Collapse

**URL:** <https://discuss.huggingface.co/t/sft-of-instruct-model-collapse/170526>\
**Category:** Models\
**Created:** [November 16, 2025, 7:58pm UTC](https://discuss.huggingface.co/t/sft-of-instruct-model-collapse/170526 "2025-11-16T19:58:23Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![entfane](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/entfane/32/48658_2.png) [@entfane](https://discuss.huggingface.co/u/entfane)\
**Post date:** [November 16, 2025, 7:58pm UTC](https://discuss.huggingface.co/t/sft-of-instruct-model-collapse/170526/1 "2025-11-16T19:58:23Z")

</div>

Has anyone also observed models collapsing if you try to SFT an Instruct model?  
This is not the first time I’m seeing this - was trying to train a model (which I thought was foundation model) and at the end the model was generating absolute gibberish and not even getting to eos token. Later found out that the model, even though was said to be ONLY pretrained, turned out to already be familiar w chat template and instruction following.  
My question - Is it a rule that if we try to sft a model that is already Instruct model that it will collapse at the end?

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 16, 2025, 11:19pm UTC](https://discuss.huggingface.co/t/sft-of-instruct-model-collapse/170526/2 "2025-11-16T23:19:26Z")

</div>

Even when using SFT based on the Instruct model, normal model weights are often generated.

However, since the Instruct model often already has its own quirks regarding special tokens and chat templates compared to the Base model, if it cannot mimic these, the result may appear broken. Also, note that older models may lack chat templates altogether or have inappropriate ones.

* * *

Short, direct answer:  
No, it is **not** a rule that SFT on an already-Instruct model will “collapse” into gibberish. Lots of people successfully SFT Instruct/chat models every day. What _does_ happen quite often is that **SFT pipelines that are slightly wrong break Instruct models much more visibly** than base models.

Your symptom:

- “absolute gibberish”
- “not even getting to eos token”

matches **technical / configuration problems** , not a fundamental “you SFTed an Instruct model, therefore it died” law.

Below I’ll put your experience in context, then connect it to concrete online reports and the real failure modes.

* * *

## 1. Background: what you did and why the model reacted badly

You thought you were fine-tuning a **pretrained/base model**.  
In reality, the checkpoint was **already instruction-tuned and chat-templated**.

That matters because:

1. A **base model** has:

2. An **Instruct/chat model** usually:

If you fine-tune it as if it were a plain LM (no template, wrong special tokens, wrong EOS handling, wrong masking, wrong tokenizer), you can:

- distort the probability distribution over tokens so that:

- and/or break things at save/load or deployment time so the weights no longer match the tokenizer.

That _looks_ like “the model collapsed because it was Instruct,” but the underlying cause is: **you violated the assumptions that Instruct model was trained under.**

* * *

## 2. Has anyone else seen this? Yes – a lot. But the causes are concrete.

Here are closely matching real reports.

### 2.1 Gemma-3-1B-IT: Instruct model → gibberish after instruction SFT

On the official Gemma 3 Instruct (`google/gemma-3-1b-it`), a user fine-tunes for instruction tasks and gets **gibberish outputs**. Google’s response:

- likely **mismatch between tokenizer and chat template** used in fine-tuning vs what the Instruct model expects,
- plus possible problems with `fp16` numerics or data formatting. ([Hugging Face](https://huggingface.co/google/gemma-3-1b-it/discussions/27 "[Gemma-3-1B] Gibberish outputs after instruction fine-tuning"))

They explicitly note that the base Gemma model (no chat special tokens) didn’t break under the same pipeline, exactly like your experience.

So: same pattern you saw, and the root cause is **format/tokenizer mismatch** , not a general rule that “instruct models collapse.”

* * *

### 2.2 Llama-2-7B-chat + softprompt: base OK, Instruct+prompt = pure gibberish

In a Hugging Face forum thread “Softprompt for Llama generating gibberish output”, a user:

- trains a soft prompt on top of `meta-llama/Llama-2-7b-chat-hf`,
- base chat model alone is fine,
- but with the softprompt attached, generations are **nonsensical character salad**. ([Hugging Face Forums](https://discuss.huggingface.co/t/softprompt-for-llama-generating-gibberish-output/153275 "Softprompt for Llama generating gibberish output - Beginners"))

The diagnosis revolves around:

- mis-handling of input formatting with the softprompt,
- incorrect placement of the prompt embeddings,
- and mismatched tokenization.

Again, it’s an **Instruct model** , and it collapses only when the **additional SFT / PEFT layer is wired wrong**.

* * *

### 2.3 GPT-2 persona-chat: base LM → fine-tune → weird chat gibberish

Older, non-instruct example, but exact same symptom:

- “Fine tuning GPT2 on persona chat dataset outputs gibberish” (HF forums). ([Hugging Face Forums](https://discuss.huggingface.co/t/fine-tuning-gpt2-on-persona-chat-dataset-outputs-gibberish/3111 "Fine tuning GPT2 on persona chat dataset outputs gibberish"))

- Training loss looks reasonable, but responses are a mess.

- Discussion points to:

This shows: you can get the **same kind of collapse without any Instruct model at all** if you mishandle the data/format.

* * *

### 2.4 Gemma-3-1B-PT: fine-tuned model OK, but gibberish after quantization

Another closely related pattern:

- User fine-tunes `google/gemma-3-1b-pt` (pretrained/base) with Unsloth + LoRA using ChatML.
- Full-precision merged model responds correctly.
- After GPTQ/AWQ/BitsAndBytes quantization, **all quantized versions produce gibberish or empty outputs**. ([Hugging Face](https://huggingface.co/google/gemma-3-1b-pt/discussions/13 "Fine-tuned Gemma-3-1b model produces gibberish/empty ..."))

Here the SFT is fine, but the **deployment step** (quantization / conversion) breaks the model, making it _look_ collapsed.

* * *

### 2.5 BART: fine-tune → gibberish until decoder is fixed

Seq2seq but the same underlying pattern:

- “What can cause model.generate (BART) output to be gibberish after fine-tuning?” (HF forums). ([Hugging Face Forums](https://discuss.huggingface.co/t/what-can-cause-model-generate-bart-output-to-be-gibberish-after-fine-tuning/934 "What can cause model.generate (BART) output to be ..."))

- After fine-tuning, `generate()` gives nonsense.

- Official answer: their decoder inputs were wrong:

- Once they fix decoding, outputs become normal.

Again: the **training isn’t inherently doomed** ; the mechanics were off.

* * *

## 3. What’s actually going on when you see “gibberish + no EOS”

Based on those reports and your description, the failure is almost always one (or a stack) of these:

1. **Tokenizer / vocab / EOS mismatch**

2. **Ignoring the model’s chat template**

3. **Over-aggressive learning rate / training schedule**

4. **Broken save/load or quantization**

5. **NaNs / numerical instability**

None of these are about “the model already being Instruct” as a fundamental problem. They are about **technical mismatch between your pipeline and the model’s assumptions**.

* * *

## 4. Why Instruct models _feel_ more fragile than base ones

This is probably the core of your intuition.

1. **More structure baked in.**  
Instruct/chat models are trained on very regular patterns:

2. **Less “room to move” without breaking behavior.**  
A base model is a general LM; you can often push it a bit and it’ll remain a fluent LM. An Instruct model has already been optimized for:

3. **But they are routinely SFTed successfully.**  
At the same time:

* * *

## 5. What this means for your rule-question

Your question:

> Is it a rule that if we try to SFT a model that is already Instruct model that it will collapse at the end?

Clear answer:

- **No, there is no such rule.**

- There is a **strong pattern** that:

But this is a consequence of **pipeline mismatch** , not an inherent property of “instructness.”

* * *

## 6. Practical takeaway (if you do this again)

To avoid repeating the collapse you saw:

1. **Always confirm what the checkpoint actually is.**

2. **Use the official SFT pattern for that family.**

3. **Keep tokenizer + model bound together.**

4. **Be gentle with hyperparameters, especially on Instruct.**

5. **Test mid-training and immediately after saving.**

If you line those up, SFT on an Instruct model is just another normal step in the stack (pretrain → SFT → preference optimization), not something that “must” end in collapse.

---

<div class="post-metadata">

**Author:** ![entfane](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/entfane/32/48658_2.png) [@entfane](https://discuss.huggingface.co/u/entfane)\
**Post date:** [November 17, 2025, 7:00am UTC](https://discuss.huggingface.co/t/sft-of-instruct-model-collapse/170526/3 "2025-11-17T07:00:39Z")

</div>

Aha! I guess I never had any luck w instruct models and managed to collapse them. Always SFTed base ones. Will check my pipelines! Thank u, @John6666

---

<div class="post-metadata">

**Author:** ![entfane](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/entfane/32/48658_2.png) [@entfane](https://discuss.huggingface.co/u/entfane)\
**Post date:** [November 21, 2025, 9:25pm UTC](https://discuss.huggingface.co/t/sft-of-instruct-model-collapse/170526/4 "2025-11-21T21:25:25Z")

</div>

@John6666 maybe u could help out - I am trying to integrate CoT into Qwen 3b so i finetuned it w Lora, 1 epoch 800 samples. Added special tokens for thinking. Loss seemed to be alright, did see a decrease and training was stable, nevertheless when testing the model it did not generate any thinking tokens and seems like started ro generate trash.  
Thinking tokens were added to embed layer and trained like this:

```py
special_tokens = ["<|think_start|>", "<|think_end|>"]
tokenizer.add_special_tokens({'additional_special_tokens': special_tokens})
model.resize_token_embeddings(len(tokenizer))
peft_config = LoraConfig(
    trainable_token_indices={'embed_tokens': tokenizer.convert_tokens_to_ids(special_tokens)},
    r = 16,
    target_modules = ["gate_proj", "down_proj", "up_proj",
                      "q_proj", "v_proj", "k_proj", "o_proj"],
    lora_alpha = 16,
    lora_dropout = 0,
    bias = "none",
    use_rslora = True,
    lora_bias = False
)

```

Any hints to what could be the issue

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 22, 2025, 12:00am UTC](https://discuss.huggingface.co/t/sft-of-instruct-model-collapse/170526/5 "2025-11-22T00:00:55Z")

</div>

You might want to take a closer look at the specifications for the special tokens and templates of the target model…?

* * *

High-level summary first:

- No, your code snippet is not obviously “illegal”, and LoRA on Qwen-3B with new thinking tokens is a reasonable idea.

- However, there are several **very specific places** this setup can go wrong:

Below is a structured walkthrough of what’s happening, what can break, and how to debug it step by step.

* * *

## 1. What you tried to do (and what PEFT expects)

You did three key things:

1. **Added new CoT markers as special tokens**

2. **Used LoRA with `trainable_token_indices`**

3. **Trained ~1 epoch on ~800 samples, loss went down, training stable.**

So structurally you’re doing what PEFT calls “train tokens alongside LoRA” with new `<think>` markers, which is a supported pattern. The fact that it then **doesn’t emit those tokens and degrades behavior** means something else in the pipeline is off.

* * *

## 2. How Qwen-3 thinks about CoT internally (context)

Qwen3 “thinking” models (e.g. Qwen3-4B / Qwen3-30B Thinking variants) use **dedicated thinking tokens** :

- Template for the last turn (from the Qwen docs):

- When you call `tokenizer.apply_chat_template(..., enable_thinking=True)`, the template inserts `<think>...</think>` around the reasoning block before the final answer. ([docs.unsloth.ai](https://docs.unsloth.ai/models/qwen3-how-to-run-and-fine-tune "Qwen3: How to Run & Fine-tune"))

For training CoT in their official pipeline, they:

- wrap reasoning content in `<think>...</think>`,
- align that with the chat template,
- and train on large math/reasoning datasets plus reasoning-aware RL. ([Qwen](https://qwenlm.github.io/blog/qwen3/ "Qwen3: Think Deeper, Act Faster | Qwen"))

You are essentially trying to replicate this behavior from scratch on Qwen-3B with **custom markers** `<|think_start|>` / `<|think_end|>` and a very small SFT dataset.

That is doable, but fragile: you need to get **all of the following right at once** :

1. Token addition + embedding training.
2. Data formatting and label masking.
3. Chat template / inference format.
4. Hyperparameters and dataset size.

If any one of those is wrong, the model can easily:

- never emit your thinking tokens, and/or
- degrade into “trash” outputs.

* * *

## 3. Main likely failure modes in your specific setup

### 3.1 The thinking tokens are never actually supervised as outputs

This is the most common issue in practice:

- You **see** `<|think_start|>` / `<|think_end|>` strings in your raw text,

- but after tokenization + collator, they may:

Typical ways this happens:

1. **SFT trainer masking everything before some “response marker”**

2. **Custom collator masking special tokens**

3. **Truncation**

Quick sanity checks:

- Take one batch from your dataloader, **before** `Trainer`:

- Confirm that:

If they do not appear in the label region, the model is never trained to generate them; they’ll almost never show up at inference.

* * *

### 3.2 Chat template mismatch: training vs inference

Qwen-style models are chat models with a specific chat template. For Qwen3, there is even an `enable_thinking` knob that changes the template to insert `<think>...</think>`. ([Qwen](https://qwen.readthedocs.io/en/latest/getting_started/concepts.html "Key Concepts - Qwen - Read the Docs"))

If you:

- train on **some custom raw format** (maybe user/assistant lines with `<|think_start|> ... <|think_end|>`),
- but at inference time, you use **`tokenizer.apply_chat_template` with the default template** ,

then the actual prompt structure the model sees at inference:

- may be completely different from what it saw during SFT,
- may not contain your markers at all,
- and may have Qwen’s original system/user/assistant tokens in places you never trained on.

This can easily look like “the model started generating trash” because:

- The prompt distribution has changed,
- The LoRA is “anchored” on patterns that never appear now,
- The sampling runs in a region of the model’s behavior that wasn’t tuned.

Concrete checks:

1. Print the **exact strings** you trained on after `apply_chat_template` (or whatever you used), and compare them to what you pass into `generate` now.

2. For the CoT runs, make sure the inference prompt **contains** your `"<|think_start|>"` and/or gives a very explicit pattern like:

3. If you’re using Qwen3’s own template, consider:

* * *

### 3.3 The new tokens’ embeddings are effectively random / not tied

You added new tokens and resized:

```python
tokenizer.add_special_tokens({'additional_special_tokens': special_tokens})
model.resize_token_embeddings(len(tokenizer))

```

By default, `resize_token_embeddings` will:

- add rows to the embedding and output head,
- initialize them randomly (unless you manually reinitialize).

The **point** of `trainable_token_indices` in LoRA is exactly to make those new rows trainable in a targeted way, without touching the rest of the embedding matrix. ([Hugging Face](https://huggingface.co/docs/peft/en/package_reference/trainable_tokens "Trainable Tokens"))

Possible issues:

1. **PEFT version not supporting `trainable_token_indices` correctly**

2. **Not actually training those embeddings**

3. **Untied LM head**

* * *

### 3.4 Dataset size and hyperparameters

You’re:

- training LoRA with new tokens,
- on ~800 samples,
- for 1 epoch.

That is extremely small compared to what Qwen uses for thinking mode (they use large math/reasoning datasets and multi-stage pipelines). ([Qwen](https://qwenlm.github.io/blog/qwen3/ "Qwen3: Think Deeper, Act Faster | Qwen"))

Even if:

- the config is correct,
- the new tokens are properly trained,
- the template matches,

you may still see:

- **no visible use of thinking tokens** in general prompts, because the model hasn’t seen enough examples to “believe” that emitting them is universally useful;
- **overall quality drop** , especially if LR was large or the dataset is noisy, because LoRA can distort behavior even with small data.

You can still experiment with small datasets, but keep expectations realistic:

- For 800 samples, you may need:

* * *

### 3.5 Qwen-specific thinking mode interactions

If you are actually using a **Qwen3 model that already supports thinking** :

- It may **already** have `<think>` and `</think>` in the tokenizer and the chat template. ([Qwen](https://qwen.readthedocs.io/en/latest/getting_started/concepts.html "Key Concepts - Qwen - Read the Docs"))

- You then add _other_ special tokens `<|think_start|>` / `<|think_end|>`, which:

In that case, a cleaner approach is:

- Use Qwen’s _existing_ `<think>` / `</think>` tokens and template.
- Fine-tune on CoT examples formatted exactly the way the docs show. ([Qwen](https://qwen.readthedocs.io/en/latest/getting_started/concepts.html "Key Concepts - Qwen - Read the Docs"))

This avoids having to maintain a parallel “shadow” thinking convention.

* * *

## 4. Concrete debugging steps

If you want a practical checklist, here is a minimal sequence that usually pinpoints the issue.

### Step 1 – Verify the tokens exist and are in labels

1. After adding special tokens and resizing, run:

2. Inspect a batch:

3. Confirm:

If they’re missing from labels, fix your masking / `response_template` / collator logic first.

* * *

### Step 2 – Verify LoRA + trainable tokens are actually active

After wrapping the model with PEFT:

```python
from peft import get_peft_model

model = get_peft_model(model, peft_config)
model.print_trainable_parameters()

```

Check that:

- `trainable parameters` is \> 0,
- embedding / trainable tokens are included (if printed in detail).

You can also do a quick gradient check for one batch:

```python
model.train()
batch = next(iter(train_dataloader))
loss = model(input_ids=batch["input_ids"], labels=batch["labels"]).loss
loss.backward()

embed = model.get_input_embeddings().weight
g = embed.grad[tokenizer.convert_tokens_to_ids("<|think_start|>")]
print(g) # should not be all zeros

```

This confirms the `<|think_start|>` row is getting gradient updates.

* * *

### Step 3 – Ensure train and inference templates match

Look at exactly what you pass to the model during training vs inference.

- If you used `apply_chat_template` with `enable_thinking=False` during training but `True` or different payloads at inference, you are not in the same regime. ([docs.unsloth.ai](https://docs.unsloth.ai/models/qwen3-how-to-run-and-fine-tune "Qwen3: How to Run & Fine-tune"))
- If you trained on raw “user:\n… assistant:\n\<|think\_start|\>…\<|think\_end|\> answer” strings, do the same at inference as a test, bypassing any chat helper code, to see if the CoT behavior appears.

* * *

### Step 4 – Compare behavior: base vs adapter

Run the **same prompt** with:

1. Base model (no LoRA / adapter disabled):
2. Fine-tuned model (LoRA enabled):

If the base is fine and the adapter model is trash, then:

- Confirm you’re loading the adapter correctly (`PeftModel.from_pretrained`, not `get_peft_model` on top of an already-adapted model). ([Hugging Face](https://huggingface.co/docs/peft/main/en/developer_guides/troubleshooting "Troubleshooting"))
- Confirm you did not accidentally save/reload in a way that dropped the resized embeddings.

* * *

## 5. A more robust recipe for your use case

If you want a safer setup for “thinking tokens with Qwen-3B”:

1. **Use Qwen’s own thinking tokens if available**

2. **If you must use `<|think_start|>` / `<|think_end|>`**

3. **Use gentle hyperparameters**

4. **Scale up data if possible**

* * *

## 6. Bottom line

- Your use of `trainable_token_indices` and added special tokens is **conceptually correct** according to PEFT docs (they even use `<think>` / `</think>` as the example). ([Hugging Face](https://huggingface.co/docs/peft/main/en/developer_guides/troubleshooting "Troubleshooting"))

- The fact that the model:

If you systematically:

- check that `<|think_start|>` / `<|think_end|>` are in labels,
- verify LoRA + trainable tokens are active,
- align training and inference templates,
- and, if possible, leverage Qwen’s native `<think>` format,

you should be able to get CoT markers working without the model collapsing into gibberish.

---

<div class="post-metadata">

**Author:** ![entfane](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/entfane/32/48658_2.png) [@entfane](https://discuss.huggingface.co/u/entfane)\
**Post date:** [November 22, 2025, 3:25pm UTC](https://discuss.huggingface.co/t/sft-of-instruct-model-collapse/170526/6 "2025-11-22T15:25:00Z")

</div>

that seems to be alright. I was checking how deepseek distilled their knowledge into qwen, they were training base model. I start to get some feeling that it might be more complicated to bake in CoT in instruct model and 800 samples might be way too little.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 22, 2025, 5:04pm UTC](https://discuss.huggingface.co/t/sft-of-instruct-model-collapse/170526/7 "2025-11-22T17:04:10Z")

</div>

Yeah. When trying to make do with 800 samples, your options are limited. Realistic approaches would be to narrow down the target tasks or the number of tensors to train (via LoRA settings, etc.), or tweak the learning rate to achieve early convergence. Alternatively, you could simply try fine-tuning on a smaller model weight.

* * *

Your feeling is essentially right, with some nuance:

- Yes: **it is harder to “bake in” CoT into an Instruct model** than into a base model, especially if you want robust, general behavior with explicit think-tags.
- Yes: **800 samples is far too little** to reliably change a 3B Instruct model’s global behavior around CoT.
- That does _not_ mean you can’t do anything useful with 800 samples, but it changes what is realistic.

I’ll break this into:

1. Why DeepSeek used base models + big data (context).
2. Why Instruct models are trickier for CoT.
3. What 800 samples can and cannot realistically achieve.
4. Feasible strategies with only ~800 samples.

* * *

## 1. What DeepSeek actually did (and why your setup is different)

DeepSeek-R1’s pipeline, in short:

- They start from a **base model** : Qwen2.5-32B, _not_ an instruct/chat model. ([arXiv](https://arxiv.org/pdf/2501.12948 "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs ..."))
- They apply **RL directly on the base model** to explore long CoT (R1-Zero), then refine to DeepSeek-R1. ([arXiv](https://arxiv.org/pdf/2501.12948 "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs ..."))
- Then they do **distillation** : R1 generates **large amounts of CoT traces** , and those are used to train smaller students (Llama, Qwen). ([Fireworks AI](https://fireworks.ai/blog/deepseek-r1-distillation-reasoning "Distillation with Reasoning: Can DeepSeek R1 Teach ..."))
- Public reporting mentions on the order of **hundreds of thousands** of distillation examples (e.g. ~800k teacher-generated traces) used to tweak other vendors’ models like Qwen and Llama. ([The Financial Times.](https://www.ft.com/content/ea803121-196f-4c61-ab70-93b38043836e "DeepSeek's 'aha moment' creates new way to build powerful AI with less money"))

So DeepSeek’s recipe is:

> big _base_ model → RL to discover CoT → **huge** teacher CoT dataset → distill into smaller/base students.

Your experiment is:

> small **Instruct** model (Qwen 3B) → LoRA → **800** CoT examples → new -like tokens.

That’s several orders of magnitude different in:

- starting point (base vs instruct),
- method (RL + distillation vs one small SFT),
- and **data scale** (hundreds of thousands vs hundreds).

Given that gap, your feeling that “my 800 samples are nowhere near DeepSeek’s regime” is absolutely correct.

* * *

## 2. Why CoT is trickier to inject into an Instruct model

### 2.1 Instruct models already have a strong policy and template

Qwen-style Instruct / chat models already:

- use a specific **chat template** (role tokens, EOS behavior),
- are tuned to answer directly and be concise,
- and in Qwen3’s case they may already support a built-in “thinking mode” with `<think>...</think>` and `enable_thinking` or `/think` / `/no_think` controls. ([Qwen](https://qwenlm.github.io/blog/qwen3/ "Qwen3: Think Deeper, Act Faster | Qwen"))

That means:

- The model has a strong prior: “user asks → assistant answers cleanly.”
- You are trying to impose a new protocol: “assistant opens `<|think_start|>`, reasons, closes `<|think_end|>`, then answers.”

With only a tiny number of examples, that new protocol is a weak signal fighting against a strong, already-learned behavior and a fixed template.

By contrast, DeepSeek’s base model did not have explicit chat/CoT behavior baked in; RL and distillation _define_ that behavior from scratch. ([arXiv](https://arxiv.org/pdf/2501.12948 "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs ..."))

### 2.2 New tokens + new behavior is a double challenge

You’re doing two things at once:

1. Introducing **brand-new tokens** (`<|think_start|>`, `<|think_end|>`).
2. Asking the model to change policy to _use them consistently_ before answers.

Training brand-new tokens to be high-probability outputs requires:

- that they appear many times in **labels** (not just inputs), and
- that the surrounding pattern is consistently rewarded.

With only 800 samples, those tokens are extremely rare compared to the rest of the vocab, and the model’s prior “answer without them” remains dominant.

Research on CoT distillation (including R1 distillation and follow-up work) assumes **large CoT corpora** to make these patterns stick. ([arXiv](https://arxiv.org/pdf/2505.24850 "Reinforcement Distillation from Teacher Data for LLM ..."))

So again, your intuition that this is harder on an Instruct model and underpowered with 800 samples is correct.

* * *

## 3. What 800 samples can and cannot realistically do

Given the scale and starting point, here’s a realistic view.

### 3.1 What 800 CoT examples **cannot** reliably do

With a 3B Instruct model and a standard LoRA setup, 800 examples is **not** enough to:

- robustly “bake in” a new global CoT behavior with explicit `<|think_*|>` tokens across tasks;
- override Qwen’s existing chat template and answer style;
- recreate anything close to DeepSeek-R1-style reasoning enhancement on diverse benchmarks.

Empirical SFT studies on 3–7B models typically use **tens of thousands** to **hundreds of thousands** of examples even for ordinary instruction tuning. ([arXiv](https://arxiv.org/html/2412.13337v1 "A Guide For Supervised Fine-Tuning Small LLMs"))

DeepSeek’s and related distillation efforts (Open-R1, curriculum distillation, etc.) operate at similar or larger scales for CoT. ([arXiv](https://arxiv.org/pdf/2501.12948 "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs ..."))

So expecting 800 samples to **globally reprogram** an Instruct model’s reasoning style is not realistic.

### 3.2 What 800 CoT examples **can** be useful for

They _can_ be useful if you narrow your ambition:

1. **Pipeline sanity check**

2. **Narrow, domain-specific behavior**

3. **Evaluation / validation set**

4. **Seed data for scaling up via a teacher**

5. **Prompt / scaffolding design**

In all of these uses, 800 samples are useful—but as **tools** (for eval, seed, or local tweaks), not as the main fuel for global CoT training.

* * *

## 4. Feasible way forward **with only these samples**

Given your constraints, here is a feasible and realistic plan.

### 4.1 Treat your 800 examples as “gold” evaluation and seed data

- Keep them mostly as **held-out eval** or **teacher seed** rather than training everything on them.

- Use them to:

### 4.2 If you still want to fine-tune on them

You can still do a **very light LoRA** SFT, but with limited expectations:

- Objective: small, domain-specific improvement; not global CoT.

- Conditions:

- Use your CoT examples only for the _domain_ where you will actually apply them (e.g. specific math tasks).

Think of this as **“adding a trick”** to the model, not redesigning its reasoning brain.

### 4.3 Get explicit CoT by switching models + prompt, not by retraining

If explicit CoT is the main goal (more than “I must use these exact tokens”):

- Use a model that already has **thinking mode** baked in, e.g. Qwen3-Thinking-4B or DeepSeek-R1-Distill-Qwen. ([Hugging Face](https://huggingface.co/Qwen/Qwen3-4B-Thinking-2507 "Qwen/Qwen3-4B-Thinking-2507"))

- Then:

This is exactly the “feasible with small data” path: start from a reasoning-oriented model and do small **domain SFT** , instead of starting from an Instruct model and trying to create reasoning behavior.

### 4.4 Use the 800 samples as a starting point to synthesize more data

If you eventually want your own CoT protocol (e.g. `<|think_start|> ... <|think_end|>` and a specific style), the realistic route is:

1. Use your 800 examples as _style anchors_.
2. Use a stronger teacher (R1, R1-distill, Qwen3-Thinking) to generate **thousands** of new CoT examples that imitate this format. ([Fireworks AI](https://fireworks.ai/blog/deepseek-r1-distillation-reasoning "Distillation with Reasoning: Can DeepSeek R1 Teach ..."))
3. Fine-tune a base or reasoning model on this larger synthetic dataset.

Your 800 examples then become the “taste” you want, not the entire diet.

* * *

## 5. Direct answer to your question

> “How about this feeling? If this feeling is correct, feasible way with only this samples?”

- Your feeling is **correct in the practical sense** :

- With only these 800 samples, a **feasible way** is:

That way, you respect what small data can and cannot do—and you align your setup much more closely with what DeepSeek-style distillation pipelines actually look like.