# How to train causal language model

**URL:** <https://discuss.huggingface.co/t/how-to-train-causal-language-model/69586>\
**Category:** 🤗Transformers\
**Created:** [January 18, 2024, 7:01am UTC](https://discuss.huggingface.co/t/how-to-train-causal-language-model/69586 "2024-01-18T07:01:27Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MarkWuICT](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@MarkWuICT](https://discuss.huggingface.co/u/MarkWuICT)\
**Post date:** [January 18, 2024, 7:01am UTC](https://discuss.huggingface.co/t/how-to-train-causal-language-model/69586/1 "2024-01-18T07:01:27Z")

</div>

I found an example in [transformers/examples/pytorch/language-modeling/run\_clm.py]  
According to the script, the trainer use dafault\_data\_collator to model causal language modelling.([transformers/examples/pytorch/language-modeling/run\_clm.py at 98dda8ed03ac3f4af5733bdddaa1dab6a81e15c1 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/98dda8ed03ac3f4af5733bdddaa1dab6a81e15c1/examples/pytorch/language-modeling/run_clm.py#L604))  
Shouldn’t we use DataCollatorForLanguageModeling to shift input and output by 1 token instead? It seems that dafault\_data\_collator can’t achieve this goal.