# Setup\_chat\_format error

**URL:** <https://discuss.huggingface.co/t/setup-chat-format-error/95582>\
**Category:** Beginners\
**Created:** [July 4, 2024, 9:15pm UTC](https://discuss.huggingface.co/t/setup-chat-format-error/95582 "2024-07-04T21:15:53Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![symonjsg](https://avatars.discourse-cdn.com/v4/letter/s/c89c15/32.png) [@symonjsg](https://discuss.huggingface.co/u/symonjsg)\
**Post date:** [July 4, 2024, 9:15pm UTC](https://discuss.huggingface.co/t/setup-chat-format-error/95582/1 "2024-07-04T21:15:53Z")

</div>

I’ve fine-tuned several models with sft using the script here ([trl/examples/scripts/sft.py at main · huggingface/trl · GitHub](https://github.com/huggingface/trl/blob/main/examples/scripts/sft.py))

However, I’ve wanted to store my data in the “conversation” style and then explicitly set the chat-format in the training pipeline, using e.g. setup\_chat\_format (from [Supervised Fine-tuning Trainer](https://huggingface.co/docs/trl/main/en/sft_trainer#add-special-tokens-for-chat-format)). But I’m having trouble using this. The sft script takes the modelname as a command line argument (see example below)

accelerate launch --config\_file examples/accelerate\_configs/multi\_gpu.yaml --num\_processes=1 examples/scripts/sft.py --model\_name MYMODEL–dataset\_name MYDATA --per\_device\_train\_batch\_size 2 --gradient\_accumulation\_steps 1 --learning\_rate 2e-4 --save\_steps 20\_000 --use\_peft --lora\_r 16 --lora\_alpha 32 --lora\_target\_modules q\_proj k\_proj v\_proj o\_proj --load\_in\_4bit --output\_dir MYOUTPUTDIR

I initially though I could just modify the sft script by adding something like the following (from [trl/docs/source/sft\_trainer.mdx at main · huggingface/trl · GitHub](https://github.com/huggingface/trl/blob/main/docs/source/sft_trainer.mdx))

from transformers import AutoModelForCausalLM, AutoTokenizer

‘’‘Load model and tokenizer’‘’  
model = AutoModelForCausalLM.from\_pretrained(MYMODEL)  
tokenizer = AutoTokenizer.from\_pretrained(MYMODEL)

‘’‘Set up the chat format with default ‘chatml’ format’‘’  
model, tokenizer = setup\_chat\_format(model, tokenizer)

But this doesn’t work. I get the following error when trying to run the resulting modified sft script.

ValueError(“You passed model\_kwargs to the SFTTrainer. But your model is already instantiated.”)

If I understand correctly the problem is my instantiating the model and applying the setup\_chat\_format to it while still having the model as a parameter of the sft script. Is there some way to get the effect of setup\_chat\_format while still just passing the model as a kwarg to the sft script? Should I use something like the formatting\_func parameter of SFTTrainer to pass a chat formatter?

Thanks in advance for any insight!