# Training a language model from scratch with tensorflow (not pytorch)? **URL:** \ **Category:** Intermediate\ **Created:** [August 5, 2021, 4:06pm UTC](https://discuss.huggingface.co/t/training-a-language-model-from-scratch-with-tensorflow-not-pytorch/9002 "2021-08-05T16:06:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\ **Post date:** [August 5, 2021, 4:06pm UTC](https://discuss.huggingface.co/t/training-a-language-model-from-scratch-with-tensorflow-not-pytorch/9002/1 "2021-08-05T16:06:41Z")
Hello there, I am interested in training a language model from scratch. Not fine tuning the usual distilbert ---- running the whole thing on my GPU instead 🕶! I found this interesting notebook [How to train a new language model from scratch using Transformers and Tokenizers](https://huggingface.co/blog/how-to-train) and I would be interested to know if there is one that uses `tensorflow` instead. I cannot have `pytorch` on my machine unfortunately. Is there a huggingface example notebook that would help me do that? Thanks! ---
**Author:** ![christopher](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/christopher/32/31595_2.png) [@christopher](https://discuss.huggingface.co/u/christopher)\ **Post date:** [August 5, 2021, 7:05pm UTC](https://discuss.huggingface.co/t/training-a-language-model-from-scratch-with-tensorflow-not-pytorch/9002/2 "2021-08-05T19:05:00Z")
Hi there! This might be what you’re after: [transformers/examples/tensorflow/language-modeling at master · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/tree/master/examples/tensorflow/language-modeling) You’d use the `run_clm` script for a GPT-2 like model, and the `run_mlm` script for a BERT-like model. EDIT: If you’re able to use docker on your machine, you could also use a [huggingface image](https://hub.docker.com/u/huggingface) to run that notebook you linked to. ---
**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\ **Post date:** [August 5, 2021, 7:52pm UTC](https://discuss.huggingface.co/t/training-a-language-model-from-scratch-with-tensorflow-not-pytorch/9002/3 "2021-08-05T19:52:02Z")
thanks @christopher aka aznavour 😃 but these notebooks are much, much more complex than the code on the notebook I liked (or the blog post). Looking for something more streamlined… ---
**Author:** ![olaffson](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/olaffson/32/2416_2.png) [@olaffson](https://discuss.huggingface.co/u/olaffson)\ **Post date:** [August 6, 2021, 12:02pm UTC](https://discuss.huggingface.co/t/training-a-language-model-from-scratch-with-tensorflow-not-pytorch/9002/4 "2021-08-06T12:02:12Z")
@christopher conceptually, I wonder if training my own language model and then fine-tune it for text-classification will work better than fine-tuning the same old distilbert model that everybody is using. The corpus I am working on is highly specialized (say, medicine for instance) so a dedicated language model makes sense. What do you think? ---
**Author:** ![christopher](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/christopher/32/31595_2.png) [@christopher](https://discuss.huggingface.co/u/christopher)\ **Post date:** [August 9, 2021, 8:55am UTC](https://discuss.huggingface.co/t/training-a-language-model-from-scratch-with-tensorflow-not-pytorch/9002/5 "2021-08-09T08:55:24Z")
I think it would depend on how much (and how different) specialized data you have (perhaps compare that to the size of the dataset the model was initially pretrained on). If it’s a considerable amount, it might make sense to continue pre-training from the checkpoint of the model you’re interested in.