# LM from Scratch for Tensorflow

**URL:** <https://discuss.huggingface.co/t/lm-from-scratch-for-tensorflow/3219>\
**Category:** 🤗Transformers\
**Created:** [January 15, 2021, 5:50pm UTC](https://discuss.huggingface.co/t/lm-from-scratch-for-tensorflow/3219 "2021-01-15T17:50:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jojo2k](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jojo2k](https://discuss.huggingface.co/u/jojo2k)\
**Post date:** [January 15, 2021, 5:50pm UTC](https://discuss.huggingface.co/t/lm-from-scratch-for-tensorflow/3219/1 "2021-01-15T17:50:57Z")

</div>

Hi,  
I was following this tutorial to train a LM from scratch: [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train)  
The result is a pytorch model. Though I need one for Tensorflow. Is there an easy way to convert it?

I tried to modify the training code by using TFTrainer, TFBertForModelLM instead. but TFTrainer is causing trouble with the Data\_collator and LineByLineTextDataset objects.  
When intitializing the trainer with the data collator i get the error: **init** () got an unexpected keyword argument ‘data\_collator’  
When calling trainer.train() (without collator) I receive the error: LineByLineTextDataset object has no attribute ‘\_variant\_tensor’ .

---

<div class="post-metadata">

**Author:** ![Jung](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jung/32/21989_2.png) [@Jung](https://discuss.huggingface.co/u/Jung)\
**Post date:** [January 16, 2021, 3:05am UTC](https://discuss.huggingface.co/t/lm-from-scratch-for-tensorflow/3219/2 "2021-01-16T03:05:01Z")

</div>

Hi! Here’s a nice example of custom TF MLM learning on XLM-Roberta with Kaggle TPU

[https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm](https://www.kaggle.com/riblidezso/finetune-xlm-roberta-on-jigsaw-test-data-with-mlm)

---

<div class="post-metadata">

**Author:** ![jojo2k](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jojo2k](https://discuss.huggingface.co/u/jojo2k)\
**Post date:** [January 18, 2021, 1:30pm UTC](https://discuss.huggingface.co/t/lm-from-scratch-for-tensorflow/3219/3 "2021-01-18T13:30:13Z")

</div>

Thank you! Will check that link later. Sometimes I wonder why I don’t generally start my research on kaggle 😃 Has been a helpful source quite a few times.