# What prompt format/template should I use for training Unsloth/Phi-3.5-mini-instruct?

**URL:** <https://discuss.huggingface.co/t/what-prompt-format-template-should-i-use-for-training-unsloth-phi-3-5-mini-instruct/177144>\
**Category:** Beginners\
**Created:** [June 25, 2026, 1:04am UTC](https://discuss.huggingface.co/t/what-prompt-format-template-should-i-use-for-training-unsloth-phi-3-5-mini-instruct/177144 "2026-06-25T01:04:28Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![mansurii](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mansurii/32/63679_2.png) [@mansurii](https://discuss.huggingface.co/u/mansurii)\
**Post date:** [June 25, 2026, 1:04am UTC](https://discuss.huggingface.co/t/what-prompt-format-template-should-i-use-for-training-unsloth-phi-3-5-mini-instruct/177144/1 "2026-06-25T01:04:28Z")

</div>

Hi, I need some help figuring out the best way to format my prompts for my dataset based on the model I’m training, and I’d appreciate some advice. Should I keep it as it is, or use a chat template? Which approach should I use, and how should I format my prompts? I’m planning to run the model using an API I’ll code. Below I have attached my current format prompt that I am using to train my Unsloth/Phi-3.5-mini-instruct Model.

````auto
Example of my current format prompt
```
from datasets import Dataset

def format_prompt(example):
    return f"### Input: {json.dumps(example['input'], ensure_ascii=False)}\n### Output: {json.dumps(example['output'], ensure_ascii=False)}<|endoftext|>"

formatted_data = [format_prompt(item) for item in file]
dataset = Dataset.from_dict({"text": formatted_data})
```
Example of my dataset input and output
```
{
    "input": {
      "insight_period": "7-Days",
      "receipts": [
        {
          "merchantName": "Lane7",
          "date": "2026-06-08",
          "total": 21.02,
          "currency": "£"
        },
        {
          "merchantName": "LVLS",
          "date": "2026-06-07",
          "total": 53.78,
          "currency": "£"
        }
      ]
    },
    "output": {
      "insights": [
        {
          "title": "Balanced Spending",
          "description": "You spent £74.80, which falls within a reasonable range. The majority of your spending came from Entertainment. There is a consistent pattern in your recent transactions."
        },
        {
          "title": "Spending Pattern",
          "description": "You made 2 transactions during this period."
        },
        {
          "title": "Category Insight",
          "description": "Your highest spending category was Entertainment."
        }
      ],
      "category": "Entertainment"
    }
  }
```

````

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 25, 2026, 5:13am UTC](https://discuss.huggingface.co/t/what-prompt-format-template-should-i-use-for-training-unsloth-phi-3-5-mini-instruct/177144/2 "2026-06-25T05:13:39Z")

</div>

Hmm… personally, when the model has one available (very old models sometimes do not) I think using the Chat Template is often the safer choice:

* * *

## Short answer

I would keep your raw `{input, output}` data structure, but I would not use the hand-written

```plaintext
### Input: ...
### Output: ...
<|endoftext|>

```

string as my default training format for `Phi-3.5-mini-instruct`.

For this model, I would start with this route instead:

1. keep your raw examples as `{input, output}`;
2. convert each row into `messages`;
3. render those `messages` with the tokenizer/model chat template;
4. use `add_generation_prompt=False` for training;
5. use the same chat-style structure again when serving the model through your API.

The key idea is to separate **how you store your dataset** from **the final text/token format the model sees**.

A minimal shape would be:

```python
import json

messages = [
    {
        "role": "system",
        "content": (
            "You convert receipt data into the requested spending insight JSON. "
            "Return only valid JSON matching the expected output shape."
        ),
    },
    {
        "role": "user",
        "content": json.dumps(example["input"], ensure_ascii=False),
    },
    {
        "role": "assistant",
        "content": json.dumps(example["output"], ensure_ascii=False),
    },
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=False,
)

```

Then use that rendered `text` in your dataset.

This is not a claim that your current `### Input / ### Output` string can never train. It probably can train as plain text-completion data. I would just treat the Phi-3.5 chat template as the lower-risk default for an already instruction-tuned chat model.

> **More detail: why I would start from the Phi-3.5 chat template**
>
> There are two different layers here:
> 
> | Layer | Example | Comment |
> | --- | --- | --- |
> | Raw data shape | `{"input": {...}, "output": {...}}` | This is fine as an internal dataset format. |
> | Rendered training text | The actual string/token sequence the model sees | This is where I would use the model chat template. |
> 
> The official Phi-3.5 sample fine-tuning script does something similar: it takes `messages`, applies
> 
> ```python
> tokenizer.apply_chat_template(
> messages,
> tokenize=False,
> add_generation_prompt=False,
> )
> 
> ```
> 
> and stores the result as a `text` field for `SFTTrainer`.
> 
> Reference:
> 
> [sample\_finetune.py · microsoft/Phi-3.5-mini-instruct at main](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/blob/main/sample_finetune.py)
> 
> The Phi-3.5 model card also shows chat-style usage with messages / roles, and the model format uses markers in the family of:
> 
> ```plaintext
> <|system|>
> ...
> <|end|>
> <|user|>
> ...
> <|end|>
> <|assistant|>
> ...
> 
> ```
> 
> Reference:
> 
> [microsoft/Phi-3.5-mini-instruct · Hugging Face](https://huggingface.co/microsoft/Phi-3.5-mini-instruct)
> 
> The general Hugging Face Transformers chat template docs explain the same idea: chat models often expect model-specific control tokens, role markers, and turn boundaries. The chat template is the tokenizer-side mechanism that converts a list of `{role, content}` messages into the final sequence.
> 
> Reference:
> 
> [Chat templates · Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating)
> 
> For training, the docs say to apply the chat template as preprocessing and usually set `add_generation_prompt=False`, because the extra assistant-start tokens used for generation are not helpful inside completed training examples.
> 
> So I would read the evidence as:
> 
> > For `Phi-3.5-mini-instruct`, a good first implementation is not a hand-written generic prompt format. It is raw data → `messages` → `tokenizer.apply_chat_template(..., add_generation_prompt=False)` → `text`.
> 
> Again, this is a default/safety recommendation, not a mathematical proof that every other format fails.

## Recommended data flow

I would structure it like this:

```plaintext
raw {input, output}
    -> messages: system / user / assistant
    -> tokenizer.apply_chat_template(...)
    -> text column
    -> SFTTrainer

```

So your current raw data can stay mostly as-is. The main change is the formatting function.

```python
def format_example(example, tokenizer):
    messages = [
        {
            "role": "system",
            "content": (
                "You convert receipt data into the requested spending insight JSON. "
                "Return only valid JSON matching the expected output shape."
            ),
        },
        {
            "role": "user",
            "content": json.dumps(example["input"], ensure_ascii=False),
        },
        {
            "role": "assistant",
            "content": json.dumps(example["output"], ensure_ascii=False),
        },
    ]

    return {
        "text": tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=False,
        )
    }

```

Before training, print one rendered example.

```python
formatted = format_example(train_data[0], tokenizer)
print(formatted["text"])

```

You want to confirm that the final string actually looks like a Phi-style chat conversation, not like a mixture of multiple prompt formats.

> **More detail: training format vs inference/API format**
>
> I would also keep training and serving as close as possible.
> 
> | Situation | Usual setting | Reason |
> | --- | --- | --- |
> | SFT training on full examples | `add_generation_prompt=False` | The assistant answer is already included in the training example. |
> | Direct Transformers generation | `add_generation_prompt=True` | The model needs the assistant turn to start. |
> | vLLM / SGLang / OpenAI-compatible serving | pass `messages` to the chat endpoint | Let the server/tokenizer apply the configured chat format. |
> 
> For inference/API use, you would normally omit the assistant answer:
> 
> ```python
> messages = [
> {
> "role": "system",
> "content": (
> "You convert receipt data into the requested spending insight JSON. "
> "Return only valid JSON matching the expected output shape."
> ),
> },
> {
> "role": "user",
> "content": json.dumps(input_payload, ensure_ascii=False),
> },
> ]
> 
> ```
> 
> If you call Transformers directly, then:
> 
> ```python
> inputs = tokenizer.apply_chat_template(
> messages,
> tokenize=True,
> add_generation_prompt=True,
> return_tensors="pt",
> )
> 
> ```
> 
> The important part is not only “which template should I train with?” but also “will my API call use the same conversation contract?”
> 
> I would avoid this mismatch:
> 
> ```plaintext
> training: custom ### Input / ### Output text
> inference: model chat template / messages
> 
> ```
> 
> and also this mismatch:
> 
> ```plaintext
> training: model chat template / messages
> inference: custom raw text prompt with different separators
> 
> ```
> 
> Either mismatch can make debugging harder. If outputs are poor, it becomes unclear whether the issue is the fine-tuning data, the template, the serving wrapper, the stop tokens, or the generation settings.

## Minimal checks before a real training run

Before starting a full job, I would inspect the tokenizer and one formatted sample.

```python
print("=== chat template ===")
print(tokenizer.chat_template)

print("=== eos token ===")
print(tokenizer.eos_token)

print("=== special tokens ===")
print(tokenizer.special_tokens_map)

print("=== rendered example ===")
print(text)

```

Things I would check:

| Check | Why |
| --- | --- |
| The rendered text contains a user turn | Confirms the input JSON is in the expected role. |
| The rendered text contains an assistant turn | Confirms the output JSON is treated as the target answer. |
| The assistant JSON is complete | Avoids training on truncated/malformed outputs. |
| Turn boundaries are consistent | Avoids mixing multiple stop/end conventions. |
| You are not manually inserting extra special tokens | Avoids EOS / stop-token confusion. |
| The same message structure can be used by the API | Avoids train/serve mismatch. |

> **More detail: special tokens, \`\<|end|\>\`, and \`\<|endoftext|\>\`**
>
> Your current string manually appends `<|endoftext|>`.
> 
> I would not start there. I would first let the tokenizer/chat template produce the separators, then inspect the rendered text.
> 
> For Phi-style chat, the visible turn boundary is usually `<|end|>` in the chat template. The Unsloth docs point to their `chat_templates.py` file as the source for supported templates:
> 
> [Chat Templates | Unsloth Documentation](https://unsloth.ai/docs/basics/chat-templates)
> 
> The current Unsloth chat template file includes Phi-family templates, and `phi-3.5` is mapped through the Phi-3 style template family:
> 
> [https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/chat\_templates.py](https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/chat_templates.py)
> 
> So I would not hand-write:
> 
> ```plaintext
> <|endoftext|>
> 
> ```
> 
> at the end of each example as the first version. Instead, I would inspect what the tokenizer already emits:
> 
> ```python
> text = tokenizer.apply_chat_template(
> messages,
> tokenize=False,
> add_generation_prompt=False,
> )
> 
> print(text)
> 
> ```
> 
> This is not a claim that manually using `<|endoftext|>` always breaks training. It is just a safer debugging path:
> 
> 1. check the tokenizer’s actual template;
> 2. check the tokenizer’s actual EOS token;
> 3. render one example;
> 4. only then decide whether you need to override anything.
> 
> Also, if you call `apply_chat_template(..., tokenize=False)` and then tokenize the string later, watch out for accidentally adding special tokens twice. The Transformers docs mention this as a general chat-template pitfall:
> 
> [Chat templates · Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating)

## If you like `### Input`, keep it inside the user message

If you prefer the readability of headings like `### Input`, I would put those inside the `user` content, not use them as the outer conversation template.

For example:

```python
messages = [
    {
        "role": "system",
        "content": "Return only the requested JSON object.",
    },
    {
        "role": "user",
        "content": (
            "### Input\n"
            + json.dumps(example["input"], ensure_ascii=False)
            + "\n\n### Task\n"
            + "Generate the spending insight JSON."
        ),
    },
    {
        "role": "assistant",
        "content": json.dumps(example["output"], ensure_ascii=False),
    },
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=False,
)

```

This keeps the outer format aligned with the model’s chat template, while still letting your task content have readable headings.

> **Optional layer: response-only / assistant-only loss**
>
> This is a separate decision from the prompt format.
> 
> There are two different questions:
> 
> | Question | Example |
> | --- | --- |
> | How do I render the example? | Chat template vs custom `### Input / ### Output`. |
> | Which tokens should contribute to loss? | Full sequence vs assistant answer only. |
> 
> I would first make sure the rendered chat text is correct. Then, if you want to train only on the assistant JSON answer and ignore the user/system tokens in the loss, look into response-only or assistant-only loss.
> 
> TRL has an `assistant_only_loss=True` option for conversational datasets:
> 
> [SFT Trainer · Hugging Face](https://huggingface.co/docs/trl/en/sft_trainer)
> 
> Important caveat: assistant-only masking depends on the chat template being able to mark assistant-generated spans. In TRL docs/issues this is often discussed in terms of `{% generation %}` / `{% endgeneration %}` markers in the chat template.
> 
> If using Unsloth’s `train_on_responses_only`, I would also verify that the labels are not fully masked.
> 
> A quick sanity check after building the trainer/dataset:
> 
> ```python
> sample = trainer.train_dataset[0]
> labels = sample["labels"]
> 
> print("non-masked labels:", (labels != -100).sum())
> 
> ```
> 
> If that prints `0`, the example is contributing no supervised loss. Possible causes include:
> 
> - the response markers do not match the rendered template;
> - the assistant part was truncated away;
> - the dataset format differs from what the masking helper expects;
> - the wrong chat template was applied.
> 
> I would not debug this first unless you are actually using response-only training. For a first pass:
> 
> 1. render with the correct chat template;
> 2. inspect the rendered string;
> 3. run a tiny training test;
> 4. only then add response-only masking if needed.

> **Optional layer: JSON reliability for an API**
>
> Since your target is an API that returns JSON, I would separate these two goals:
> 
> | Goal | Where it belongs |
> | --- | --- |
> | Teach the model the desired input/output mapping | SFT data and prompt/template design |
> | Ensure valid/schema-compliant JSON in production | Serving logic, validation, retries, constrained decoding, evals |
> 
> SFT can teach the model to produce JSON-shaped answers. But if the API must reliably return valid JSON, I would still consider:
> 
> - JSON validation after generation;
> - retry or repair logic;
> - low temperature for extraction/conversion-style tasks;
> - `max_new_tokens` large enough for the whole JSON answer;
> - correct stop token handling;
> - structured/constrained decoding if your serving stack supports it.
> 
> For example, vLLM has structured output support:
> 
> [Structured Outputs - vLLM](https://docs.vllm.ai/en/latest/features/structured_outputs/)
> 
> This is not required to answer the prompt-format question, but it is relevant if the API needs to be robust rather than just “usually returns JSON-looking text.”

## Practical summary

My first implementation would be:

```python
import json

def format_example(example, tokenizer):
    messages = [
        {
            "role": "system",
            "content": (
                "You convert receipt data into the requested spending insight JSON. "
                "Return only valid JSON matching the expected output shape."
            ),
        },
        {
            "role": "user",
            "content": json.dumps(example["input"], ensure_ascii=False),
        },
        {
            "role": "assistant",
            "content": json.dumps(example["output"], ensure_ascii=False),
        },
    ]

    return {
        "text": tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=False,
        )
    }

```

Then inspect before training:

```python
formatted = format_example(train_data[0], tokenizer)
print(formatted["text"])

```

And for API inference, use the same message structure, but without the assistant answer:

```python
messages = [
    {
        "role": "system",
        "content": (
            "You convert receipt data into the requested spending insight JSON. "
            "Return only valid JSON matching the expected output shape."
        ),
    },
    {
        "role": "user",
        "content": json.dumps(input_payload, ensure_ascii=False),
    },
]

```

If using Transformers directly:

```python
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
)

```

So my practical recommendation would be:

- keep `{input, output}` as the raw data format;
- convert rows into `messages`;
- use the Phi-3.5 tokenizer chat template for the final training text;
- use `add_generation_prompt=False` for training;
- use the same chat structure for API inference;
- do not manually append `<|endoftext|>` until you have inspected what the tokenizer already emits;
- treat response-only loss and production JSON validation as separate follow-up layers.

> **References**
>
> Main references I would check:
> 
> - Phi-3.5-mini-instruct model card  
> [microsoft/Phi-3.5-mini-instruct · Hugging Face](https://huggingface.co/microsoft/Phi-3.5-mini-instruct)
> 
> - Phi-3.5 sample fine-tuning script  
> [sample\_finetune.py · microsoft/Phi-3.5-mini-instruct at main](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/blob/main/sample_finetune.py)
> 
> - Transformers chat template docs  
> [Chat templates · Hugging Face](https://huggingface.co/docs/transformers/en/chat_templating)
> 
> - TRL SFTTrainer docs  
> [SFT Trainer · Hugging Face](https://huggingface.co/docs/trl/en/sft_trainer)
> 
> - Unsloth chat template docs  
> [Chat Templates | Unsloth Documentation](https://unsloth.ai/docs/basics/chat-templates)
> 
> - Unsloth chat template code  
> [https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/chat\_templates.py](https://raw.githubusercontent.com/unslothai/unsloth/main/unsloth/chat_templates.py)
> 
> - vLLM structured outputs, if this becomes a production JSON API question  
> [Structured Outputs - vLLM](https://docs.vllm.ai/en/latest/features/structured_outputs/)

---

<div class="post-metadata">

**Author:** ![endlessicingz](https://avatars.discourse-cdn.com/v4/letter/e/d2c977/32.png) [@endlessicingz](https://discuss.huggingface.co/u/endlessicingz)\
**Post date:** [July 5, 2026, 6:31am UTC](https://discuss.huggingface.co/t/what-prompt-format-template-should-i-use-for-training-unsloth-phi-3-5-mini-instruct/177144/3 "2026-07-05T06:31:13Z")

</div>

Wow, John. That is a generous share inclining me to see value in this community [despite other experiences](https://discuss.huggingface.co/t/self-aggrandizing-poster-flags-all-dissenting-comments/177463/1).