# How data should be structured to Fine-Tune a CausalLM

**URL:** <https://discuss.huggingface.co/t/how-data-should-be-structured-to-fine-tune-a-causallm/94198>\
**Category:** 🤗Transformers\
**Created:** [June 26, 2024, 12:15pm UTC](https://discuss.huggingface.co/t/how-data-should-be-structured-to-fine-tune-a-causallm/94198 "2024-06-26T12:15:38Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![NevadaM](https://avatars.discourse-cdn.com/v4/letter/n/e99b99/32.png) [@NevadaM](https://discuss.huggingface.co/u/NevadaM)\
**Post date:** [June 26, 2024, 12:15pm UTC](https://discuss.huggingface.co/t/how-data-should-be-structured-to-fine-tune-a-causallm/94198/1 "2024-06-26T12:15:38Z")

</div>

Hi all,

Here’s a scenario: I want to fine-tune GPT-2 on the English subset of wikimedia/wikipedia. This dataset is, famously, a collections of blocks of text that looks a bit like figure 1 below.

I’m going to be fine-tuning using TensorFlow (just because I need to use it for a downstream experiment afterward). Because GPT-2 is autoregressive, I’m assuming the data should therefore look a bit like figure 2 (only tokenized, batched and shuffled, padded etc. ).

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/e/1/e1294267a5e5130e1ec2510b006e95cc9ef3bf11.png)

Is this assumption correct? There’s no documentation anywhere on this question, and I guess my reasoning is that the output of a causalLM is its predicted logits for the token after input. This data structure therefore plays into sparse categorical loss funcs and, importantly, is uniform/rectangular in shape.

If this assumption is wrong, I can’t visualise what else could be the case. Examples of fine-tuning instruct models seem to have no labels at all, rather just fitting the model over a set of input\_ids (see: [https://ai.google.dev/gemma/docs/lora\_tuning#load\_dataset](https://ai.google.dev/gemma/docs/lora_tuning#load_dataset)), but I tested this with my implementation and the error was something along the lines of “there was no label column detected in the dataset” - makes sense, given all that was going in was input\_ids and an attention mask.

This topic is a real gap in the documentation imo, but I might just be thinking too hard about it haha. Any help would be appreciated.

NM

---

<div class="post-metadata">

**Author:** ![MattiLinnanvuori](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MattiLinnanvuori](https://discuss.huggingface.co/u/MattiLinnanvuori)\
**Post date:** [June 28, 2024, 6:58am UTC](https://discuss.huggingface.co/t/how-data-should-be-structured-to-fine-tune-a-causallm/94198/2 "2024-06-28T06:58:26Z")

</div>

[Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) See the documentation in the link for guidance of fine-tuning causal language modelling.