# SFTTrainerflags blocks assistant\_only\_loss=True

**URL:** <https://discuss.huggingface.co/t/sfttrainerflags-blocks-assistant-only-loss-true/176210>\
**Category:** 🤗Transformers\
**Created:** [May 25, 2026, 8:04am UTC](https://discuss.huggingface.co/t/sfttrainerflags-blocks-assistant-only-loss-true/176210 "2026-05-25T08:04:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kegintheai](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kegintheai/32/59166_2.png) [@kegintheai](https://discuss.huggingface.co/u/kegintheai)\
**Post date:** [May 25, 2026, 8:04am UTC](https://discuss.huggingface.co/t/sfttrainerflags-blocks-assistant-only-loss-true/176210/1 "2026-05-25T08:04:06Z")

</div>

Hi. I came across this interesting case. I’m fine-tuning Qwen3.5-4B-Base for language-modeling only (no multi-model). I’m using AutoTokenizer instead of AutoProcessor. I updated the tokenizer\_chat\_template to add the {%generation%} tag so I can use assistant\_only\_loss=True

When I set assistant\_only\_loss=True, I get the following objection from SFTTrainer:

**ValueError: Assistant-only loss is not yet supported for vision-language models. Please set `assistant_only_loss=False` in the `SFTConfig`.**

The justification I got is: _“It seems like the error is coming from `SFTTrainer` itself — it inspects the model’s `config` to detect if it’s a VLM. Qwen3.5 has vision capabilities registered in its config, so even if you’re using it as a language-only model, `SFTTrainer`flags it as a VLM and blocks `assistant_only_loss=True`.”_

I wonder if there’s a workaround for SFTTrainer to treat the model as language only since I’m defining it as: from transformers import AutoModelForCausalLM.

Thanks!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [May 25, 2026, 10:22am UTC](https://discuss.huggingface.co/t/sfttrainerflags-blocks-assistant-only-loss-true/176210/2 "2026-05-25T10:22:56Z")

</div>

It looks like this is falling into a gap between supported paths:

* * *

I think your read is basically right, but I would frame it slightly differently:

This does not look like a simple user mistake. It looks like a boundary case between:

1. a model artifact that is published/handled as a VLM,
2. a user workflow that is intentionally text-only,
3. TRL’s current `SFTTrainer` VLM path, and
4. the still-fragile implementation details behind `assistant_only_loss=True`.

The short version is:

> `assistant_only_loss=True` is supported for the text/conversational SFT path when assistant-token masks can be produced from the chat template, but TRL currently blocks it for VLMs. Since `Qwen/Qwen3.5-4B-Base` resolves through the VLM/processor path, this check is triggered even if your dataset is text-only.

That makes the error understandable, but the UX is confusing.

## Why this happens

The model page for [`Qwen/Qwen3.5-4B-Base`](https://huggingface.co/Qwen/Qwen3.5-4B-Base) is not presented like a plain text-only Causal LM path. The Hub usage panel shows `image-text-to-text`, `AutoProcessor`, and `AutoModelForImageTextToText` usage for this artifact:

- [`Qwen/Qwen3.5-4B-Base` model card](https://huggingface.co/Qwen/Qwen3.5-4B-Base)
- [`Qwen/Qwen3.5-4B-Base` Transformers usage snippet](https://huggingface.co/Qwen/Qwen3.5-4B-Base/tree/main)

So even if your training data is text-only, the model artifact itself is still a vision-language capable artifact from the library/tooling point of view.

On the TRL side, the important current signal is this tracking issue:

- [`huggingface/trl#5471` — Add `{% generation %}` chat templates for common model families](https://github.com/huggingface/trl/issues/5471)

That issue explicitly lists VLM families, including Qwen3.5, and notes:

> VLMs currently don’t support `assistant_only_loss` in SFT (blocked by a separate check).

So I would not treat this as a random failure. It is a known unsupported combination.

## Why `assistant_only_loss=True` is more fragile than it looks

`assistant_only_loss=True` is not just a boolean that says “ignore user messages.” It depends on a whole chain working correctly:

- the dataset must be in the expected conversational format;
- the chat template must support assistant-token masking;
- the template needs `{% generation %}` / `{% endgeneration %}` markers;
- the tokenizer/processor must return a correct assistant mask;
- truncation must not remove the supervised assistant span;
- the data collator must actually apply that mask to `labels`;
- and in VLMs, image placeholders / image tokens / processor-specific expansion must not shift or break the mask.

The TRL docs describe the text/conversational path here:

- [TRL SFT Trainer docs — train on assistant messages only](https://huggingface.co/docs/trl/en/sft_trainer#train-on-assistant-messages-only)
- [`SFTConfig.assistant_only_loss`](https://github.com/huggingface/trl/blob/main/trl/trainer/sft_config.py)

The Transformers docs describe the lower-level assistant-mask mechanism here:

- [Transformers tokenizer docs — `return_assistant_tokens_mask`](https://huggingface.co/docs/transformers/main_classes/tokenizer)
- [Transformers chat templates](https://huggingface.co/docs/transformers/chat_templating)
- [Transformers multimodal chat templates](https://huggingface.co/docs/transformers/en/chat_templating_multimodal)
- [Transformers processors](https://huggingface.co/docs/transformers/en/main_classes/processors)

There are also related issues showing why this area is brittle, especially once `ProcessorMixin` / multimodal paths are involved:

- [`huggingface/transformers#36713` — `return_assistant_tokens_mask` argument is blocked in `ProcessorMixin.apply_chat_template`](https://github.com/huggingface/transformers/issues/36713)
- [`huggingface/transformers#44521` — `apply_chat_template` returns all-zero `assistant_masks` for multimodal input](https://github.com/huggingface/transformers/issues/44521)
- [`huggingface/trl#3751` — VLM SFT computes loss for the whole sequence, including prompt/user content](https://github.com/huggingface/trl/issues/3751)

That is why the TRL check is conservative. Allowing `assistant_only_loss=True` on the VLM path without correct masking would be worse than raising an error, because it could silently train on the wrong tokens.

## What I would try first for a strictly text-only experiment

If your workload is truly text-only, I would first try to force the text/tokenizer path by explicitly passing the tokenizer as `processing_class`.

For example:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from trl import SFTTrainer, SFTConfig

model_id = "Qwen/Qwen3.5-4B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(
        output_dir="<your-output-dir>",
        assistant_only_loss=True,
        # other args...
    ),
    train_dataset=train_dataset,
    processing_class=tokenizer,
)

```

The important part is not just creating an `AutoTokenizer`. The important part is passing it into `SFTTrainer`:

```python
processing_class=tokenizer

```

Otherwise, `SFTTrainer` may resolve the default processing class through the processor path, and the model is then treated as a VLM.

However, I would treat this as a workaround, not as proof that VLM + `assistant_only_loss` is officially supported.

After doing this, I would always inspect the labels before training:

```python
batch = trainer.data_collator.torch_call([train_dataset[0]])
labels = batch["labels"][0]

print("supervised tokens:", (labels != -100).sum().item())
print(tokenizer.decode(labels[labels != -100]))

```

You want the decoded supervised text to contain only the assistant answer span. If system/user/prompt text appears there, then the loss mask is not doing what you think it is doing.

This check is important because “the trainer runs” and “the correct tokens receive loss” are not the same thing.

## What I would prefer for a more robust text-only setup

If the actual goal is:

> train only on the answer/completion, not on the prompt/user/system text

then I would prefer a prompt-completion dataset and `completion_only_loss=True`, instead of relying on assistant masks from a chat template.

That path is conceptually simpler because the supervised boundary is explicit in the dataset schema:

```python
{
    "prompt": "<|im_start|>system\n...\n<|im_end|>\n<|im_start|>user\n...\n<|im_end|>\n<|im_start|>assistant\n",
    "completion": "The assistant answer goes here.<|im_end|>"
}

```

Then configure SFT like this:

```python
from trl import SFTTrainer, SFTConfig

trainer = SFTTrainer(
    model=model,
    args=SFTConfig(
        output_dir="<your-output-dir>",
        completion_only_loss=True,
        assistant_only_loss=False,
    ),
    train_dataset=prompt_completion_dataset,
    processing_class=tokenizer,
)

```

Relevant docs:

- [TRL SFT Trainer — dataset formats](https://huggingface.co/docs/trl/en/sft_trainer#dataset-format-support)
- [TRL SFT Trainer — train on completion only](https://huggingface.co/docs/trl/en/sft_trainer#train-on-completion-only)
- [`SFTConfig.completion_only_loss`](https://github.com/huggingface/trl/blob/main/trl/trainer/sft_config.py)

For this particular case, I would consider `completion_only_loss=True` the more durable route if you do not actually need multimodal inputs.

## If you really want to train it as a VLM

If you are actually using images/multimodal messages, I would not currently expect `assistant_only_loss=True` to work through the normal TRL SFT path.

The relevant upstream signal is still:

- [`huggingface/trl#5471`](https://github.com/huggingface/trl/issues/5471): VLMs currently do not support `assistant_only_loss` in SFT.
- [`huggingface/trl#3751`](https://github.com/huggingface/trl/issues/3751): VLM SFT examples currently raise the question of masking out prompt/user content.

In that case, the choices are roughly:

1. accept whole-sequence loss for now;
2. restructure into prompt-completion if your data shape allows it;
3. write a custom collator and verify labels very carefully;
4. wait for upstream VLM assistant-mask support.

A custom collator is possible, but I would be careful. For VLMs, masking is not just “find assistant text and set everything else to `-100`.” You also need to account for processor output, image placeholders, image tokens, multimodal token expansion, truncation, special tokens, and model-specific label handling.

## What I think TRL could improve

The current error is technically reasonable:

```plaintext
Assistant-only loss is not yet supported for vision-language models.

```

But for this specific failure mode, it does not explain why a text-only user hit a VLM error.

A more helpful error would say something like:

```plaintext
This model is being treated as a VLM because the resolved processing class is a ProcessorMixin/AutoProcessor path. VLMs do not currently support assistant_only_loss in SFT. If your workload is strictly text-only, try passing a tokenizer explicitly as processing_class=tokenizer, or use a prompt-completion dataset with completion_only_loss=True.

```

That would make the workaround and the distinction much clearer.

## My practical recommendation

For a quick experiment:

```python
processing_class=tokenizer

```

then inspect `labels != -100`.

For a more reliable text-only fine-tuning setup:

```python
completion_only_loss=True
assistant_only_loss=False

```

with an explicit prompt-completion dataset.

For actual multimodal/VLM SFT:

```python
assistant_only_loss=False

```

unless you are prepared to write and validate a custom collator.

So overall: I think your intuition is right that this is a “gap” rather than a simple user error. But given the current TRL issue tracker and docs, I would treat the current behavior as an intentional unsupported-path guard, not as a straightforward bug.

---

<div class="post-metadata">

**Author:** ![kegintheai](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kegintheai/32/59166_2.png) [@kegintheai](https://discuss.huggingface.co/u/kegintheai)\
**Post date:** [May 25, 2026, 6:24pm UTC](https://discuss.huggingface.co/t/sfttrainerflags-blocks-assistant-only-loss-true/176210/3 "2026-05-25T18:24:03Z")

</div>

Hi @John6666 thank you so much for your truly valuable feedback as always. I really appreciate it!

Regarding the action items you mentioned:

- **force the text/tokenizer path by explicitly passing the tokenizer as `processing_class`:**

I’m currently doing it. However, at first, I was not directly passing the tokenizer to the trainer as an SFTTrainer parameter. Instead, I was passing it to the data\_collator which is then passed to the trainer:

_from transformers import DataCollatorForLanguageModeling_  
_data\_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)_

trainer = SFTTrainer(  
model=model,  
train\_dataset=train\_dataset,  
eval\_dataset=eval\_dataset,  
data\_collator=data\_collator,  
args=sft\_config,  
peft\_config=peft\_config,  
)

After I read your message, I passed it to the sfttrainer and the error disappeared:

trainer = SFTTrainer(  
model=model,  
train\_dataset=train\_dataset,  
eval\_dataset=eval\_dataset,  
processing\_class=tokenizer,  
data\_collator=data\_collator,  
args=sft\_config,  
peft\_config=peft\_config,  
)

I will trade the buff output tomorrow to check if any masking took place and I’ll update you. Thanks!

---

<div class="post-metadata">

**Author:** ![kegintheai](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kegintheai/32/59166_2.png) [@kegintheai](https://discuss.huggingface.co/u/kegintheai)\
**Post date:** [May 26, 2026, 4:10pm UTC](https://discuss.huggingface.co/t/sfttrainerflags-blocks-assistant-only-loss-true/176210/4 "2026-05-26T16:10:52Z")

</div>

Hi @John6666! assistant\_only\_loss=True is working well now. Thank you so much for your great support! I really appreciate it!