# Data format for fine-tune base model

**URL:** <https://discuss.huggingface.co/t/data-format-for-fine-tune-base-model/144563>\
**Category:** Intermediate\
**Created:** [March 7, 2025, 5:52am UTC](https://discuss.huggingface.co/t/data-format-for-fine-tune-base-model/144563 "2025-03-07T05:52:49Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![AKeigo](https://avatars.discourse-cdn.com/v4/letter/a/3e96dc/32.png) [@AKeigo](https://discuss.huggingface.co/u/AKeigo)\
**Post date:** [March 7, 2025, 5:52am UTC](https://discuss.huggingface.co/t/data-format-for-fine-tune-base-model/144563/1 "2025-03-07T05:52:49Z")

</div>

I want to fine-tune a base model for single purpose (write a story).

There’re two ways to design the data:

1. {prompt}\<special\_token\>{story}
2. You are a creative writing assistant. Please write a story accoarding to the prompt. Here is the prompt: {prompt}\<special\_token\>{story}

Which one should I pick and why?  
(I did an experiment, and it shows #1 is better. But I still a little curious about this.)

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 7, 2025, 3:44pm UTC](https://discuss.huggingface.co/t/data-format-for-fine-tune-base-model/144563/2 "2025-03-07T15:44:27Z")

</div>

I tried searching with the help of [Hugging Chat](https://huggingface.co/chat/), but it seems that the 1 format is the best.

If you’re curious, why not take a look at [the actual data sets on Hugging Face](https://huggingface.co/datasets?sort=trending&search=story)?  
Well, many people process the data just before training the model, so it might be a bit messy at the data set stage…😅

* * *

**Answer:**

When fine-tuning a base model for story writing using the Hugging Face Trainer ecosystem, the recommended approach is to use the first dataset structure: `<prompt><special_token><story>`. Here’s a structured explanation of why this is the best choice:

1. **Clarity and Focus:**

2. **Efficiency:**

3. **Alignment with Best Practices:**

4. **Single-Purpose Task:**

5. **Flexibility for Future Tasks:**

In conclusion, the first dataset structure offers the optimal balance of clarity, efficiency, and alignment with best practices for fine-tuning within the Hugging Face Trainer ecosystem, particularly for a focused task like story writing.

---

<div class="post-metadata">

**Author:** ![AKeigo](https://avatars.discourse-cdn.com/v4/letter/a/3e96dc/32.png) [@AKeigo](https://discuss.huggingface.co/u/AKeigo)\
**Post date:** [March 10, 2025, 1:08am UTC](https://discuss.huggingface.co/t/data-format-for-fine-tune-base-model/144563/3 "2025-03-10T01:08:18Z")

</div>

Thanks for helping 🙂

The experiment I ran has some bug, so I did some other tests.

Here’s my obersavation and guesses:

1. I fine-tune Llama-3.1-base 8B on 10k dataset.
2. On the new experiment, #2 starts with a litter smaller loss, but at the end of training, #1 and #2 has almost the same loss.
3. So, I think both formats is OK. Personally, I think #1 is more elegant. But if you have a brilliant system prompt, or don’t have enough data, then #2 is worth to try.