# SFTTrainer work but without result

**URL:** <https://discuss.huggingface.co/t/sfttrainer-work-but-without-result/169118>\
**Category:** Beginners\
**Created:** [October 13, 2025, 8:54am UTC](https://discuss.huggingface.co/t/sfttrainer-work-but-without-result/169118 "2025-10-13T08:54:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![companyit0dev](https://avatars.discourse-cdn.com/v4/letter/c/ecd19e/32.png) [@companyit0dev](https://discuss.huggingface.co/u/companyit0dev)\
**Post date:** [October 13, 2025, 8:54am UTC](https://discuss.huggingface.co/t/sfttrainer-work-but-without-result/169118/1 "2025-10-13T08:54:53Z")

</div>

I have jsonl 100 messages with system(You are assistent), user(question) and assistent.

```auto
from trl import SFTTrainer
from datasets import load_dataset
from peft import LoraConfig
from transformers import (
    AutoModelForCausalLM,
    BitsAndBytesConfig
)

dataset = load_dataset('json', data_files='dataset.jsonl', split="train")
peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.1",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True),
    device_map="auto",
)

trainer = SFTTrainer(
    model,
    train_dataset=dataset,
    peft_config=LoraConfig()
)

trainer.train()
trainer.model.save_pretrained("mistral-sft-adapter")
trainer.tokenizer.save_pretrained("mistral-ft-adapter")

```

but when I test the chat, the answers of the model are not at all similar to the training data. Why? I do something wrong or my dataset very small? PS run via accelerate on nvidia 12Gb

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [October 13, 2025, 9:41am UTC](https://discuss.huggingface.co/t/sfttrainer-work-but-without-result/169118/2 "2025-10-13T09:41:24Z")

</div>

Seems some mistakes?

* * *

Your results don’t reflect training because you (1) didn’t train or infer with Mistral’s chat template, (2) computed loss on the whole string instead of only the assistant spans, (3) passed a fresh empty `LoraConfig()` to `SFTTrainer` instead of your tuned one, and (4) likely tested the **base** model without loading your LoRA adapter. One more factor: 100 samples is very small for steering a 7B instruct model unless you intentionally overfit. ([Hugging Face](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 "mistralai/Mistral-7B-Instruct-v0.1 · Hugging Face"))

## What to fix, concretely

1. **Use the Mistral chat template for both training and inference.**  
Mistral-Instruct expects `[INST] ... [/INST]`. Build training texts with `tokenizer.apply_chat_template(...)`. Do the same at inference. If you skip this, the model sees the wrong control tokens and ignores your patterns. Docs show the exact format and an `apply_chat_template` example. (Accessed Oct 13, 2025.) ([Hugging Face](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 "mistralai/Mistral-7B-Instruct-v0.1 · Hugging Face"))

2. **Train on the assistant’s tokens only.**  
Otherwise the model learns to copy the prompt. In TRL you can either:

- set `assistant_only_loss=True` (newer TRL), or
- use `DataCollatorForCompletionOnlyLM(response_template='[/INST] ')` to mask labels before the assistant span. (Mind the space after `]`.) (Accessed Oct 13, 2025.) ([Hugging Face](https://huggingface.co/docs/trl/v0.20.0/en/sft_trainer "SFT Trainer"))

1. **Pass the correct LoRA config.**  
You created `peft_config = LoraConfig(...)` and then threw it away by calling `peft_config=LoraConfig()`. Pass the one you defined.

2. **Load the adapter at inference.**  
Saving only the adapter means the base model’s behavior is unchanged until you attach the adapter with `PeftModel.from_pretrained(...)` or `model.load_adapter(...)`. ([Hugging Face](https://huggingface.co/docs/transformers/en/peft "PEFT"))

3. **Small dataset ⇒ overfit on purpose or add data.**  
100 chats rarely shift a strong 7B instruct model. Either run many epochs with a higher adapter LR to overfit as a sanity check or scale to thousands of samples. QLoRA helps with VRAM, not data needs. (Accessed Oct 13, 2025.) ([arXiv](https://arxiv.org/abs/2305.14314 "QLoRA: Efficient Finetuning of Quantized LLMs"))

* * *

## Minimal corrected script (QLoRA + proper masking + correct LoRA + template)

```python
# docs:
# - Mistral template: https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 # [INST] ... [/INST]
# - Chat templates: https://huggingface.co/docs/transformers/en/chat_templating
# - TRL SFTTrainer + loss masking: https://huggingface.co/docs/trl/en/sft_trainer
# - Load LoRA adapter for inference: https://huggingface.co/docs/transformers/en/peft

from datasets import load_dataset
from transformers import (AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig,
                          TrainingArguments)
from trl import SFTTrainer, DataCollatorForCompletionOnlyLM, SFTConfig
from peft import LoraConfig, PeftModel
import torch

model_id = "mistralai/Mistral-7B-Instruct-v0.1"

tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)

# Map your JSONL {system, user, assistant} into a single "text" column using the chat template
def row_to_text(ex):
    msgs = []
    if ex.get("system"):
        msgs.append({"role": "system", "content": ex["system"]})
    msgs.append({"role": "user", "content": ex["user"]})
    msgs.append({"role": "assistant", "content": ex["assistant"]})
    return {"text": tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False)}

ds = load_dataset("json", data_files="dataset.jsonl", split="train")
ds = ds.map(row_to_text, remove_columns=ds.column_names)

bnb4 = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True, # QLoRA trick
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

base = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb4, device_map="auto")

peft_cfg = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM"
)

# Mask labels so loss is computed only on the assistant completion
collator = DataCollatorForCompletionOnlyLM(response_template="[/INST] ", tokenizer=tok)

# Intentionally overfit small data to verify signal
train_args = TrainingArguments(
    output_dir="out",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=16,
    num_train_epochs=6,
    learning_rate=5e-5, # higher LR for adapters on tiny data
    bf16=True,
    logging_steps=5,
    save_strategy="epoch",
)

sft_cfg = SFTConfig( # if your TRL is recent, you may set assistant_only_loss=True instead of the collator
    dataset_text_field="text",
    packing=False,
)

trainer = SFTTrainer(
    model=base,
    processing_class=tok,
    train_dataset=ds,
    data_collator=collator,
    peft_config=peft_cfg, # <-- use the correct config
    args=train_args,
    **sft_cfg.to_dict(),
)

trainer.train()

# Save adapter + tokenizer together
trainer.model.save_pretrained("mistral-sft-adapter")
tok.save_pretrained("mistral-sft-adapter")

# -------- Inference (attach adapter and use SAME template) --------
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb4, device_map="auto")
model = PeftModel.from_pretrained(base, "mistral-sft-adapter").eval()

messages = [
  {"role": "system", "content": "You are assistant."},
  {"role": "user", "content": "Ask me the *exact* question you trained on, to sanity-check."}
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7, top_p=0.9)
print(tok.decode(out[0], skip_special_tokens=True))

```

Why each element matters:

- **Template** ensures `[INST] ... [/INST]` framing matches pretraining. ([Hugging Face](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1 "mistralai/Mistral-7B-Instruct-v0.1 · Hugging Face"))
- **Masking** ensures cross-entropy hits only assistant tokens. Use `assistant_only_loss=True` (new TRL) or the collator. ([Hugging Face](https://huggingface.co/docs/trl/v0.20.0/en/sft_trainer "SFT Trainer"))
- **QLoRA config** fits 7B on 12 GB VRAM with NF4 and bf16 compute. ([Hugging Face](https://huggingface.co/docs/transformers/v4.51.1/quantization/bitsandbytes "bitsandbytes"))
- **Adapter loading** is required at inference or your outputs come from the base model. ([Hugging Face](https://huggingface.co/docs/transformers/en/peft "PEFT"))