# Missing info in LLM Course?

**URL:** <https://discuss.huggingface.co/t/missing-info-in-llm-course/174258>\
**Category:** Course\
**Created:** [March 14, 2026, 2:13am UTC](https://discuss.huggingface.co/t/missing-info-in-llm-course/174258 "2026-03-14T02:13:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Roman190928](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/roman190928/32/58617_2.png) [@Roman190928](https://discuss.huggingface.co/u/Roman190928)\
**Post date:** [March 14, 2026, 2:13am UTC](https://discuss.huggingface.co/t/missing-info-in-llm-course/174258/1 "2026-03-14T02:13:52Z")

</div>

I know the libraries hf provides are nice, but in the cources ive looked at… are somewhat lacking. LLM course doesnt mention training from scratch? ive checked it with ctrl + f, then searching it for “train”

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 14, 2026, 5:42am UTC](https://discuss.huggingface.co/t/missing-info-in-llm-course/174258/2 "2026-03-14T05:42:27Z")

</div>

Is this the closest match within the LLM course? [causal language modeling pretraining](https://huggingface.co/learn/llm-course/chapter7/6).

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [March 14, 2026, 6:09am UTC](https://discuss.huggingface.co/t/missing-info-in-llm-course/174258/3 "2026-03-14T06:09:26Z")

</div>

If I put together a rough collection using online resources, something like this:

* * *

The Hugging Face LLM Course does include tokenizer training and a **scaled-down** causal-LM-from-scratch example. The deeper large-scale training material sits elsewhere, especially in HF’s training handbook/playbook, while **CS336** is the closest thing to a full “language modeling from scratch” course. ([Hugging Face](https://huggingface.co/learn/llm-course/chapter6/1 "Introduction"))

## Track 1 — Understand what you are training

**Use this track when:**  
Transformers still feel blurry. You can run code, but you do not yet have a clean mental model of tokenization, next-token prediction, attention, and causal LM training.

**Goal:**  
Understand the pipeline end to end before caring about scale.

**Read in this order:**

1. **The Illustrated Transformer**  
Best first stop. It is a visual introduction meant to simplify the Transformer concepts step by step. ([jalammar.github.io](https://jalammar.github.io/illustrated-transformer/ "The Illustrated Transformer"))

2. **The Annotated Transformer**  
Best second stop. It is the implementation-oriented bridge from the paper to working code. ([nlp.seas.harvard.edu](https://nlp.seas.harvard.edu/2018/04/03/attention.html "The Annotated Transformer"))

3. **CS336: Language Modeling from Scratch**  
Best full course for the whole stack: tokenization, data, model construction, training, and evaluation. ([Stanford CS336](https://cs336.stanford.edu/ "Stanford CS336 | Language Modeling from Scratch"))

4. **HF Chapter 6**  
Use this to learn tokenization properly. HF explicitly states that training a tokenizer is **not** the same as training a model, and Chapter 6 is about training a brand new tokenizer so it can later be used to pretrain a language model. ([Hugging Face](https://huggingface.co/learn/llm-course/chapter6/2 "Training a new tokenizer from an old one"))

5. **HF Chapter 7.6**  
Then do one small from-scratch causal LM run in the HF style. HF describes this section as training a completely new model from scratch on a Python code corpus with `Trainer` and Accelerate. ([Hugging Face](https://huggingface.co/learn/llm-course/chapter7/6 "Training a causal language model from scratch"))

**You are done with Track 1 when:**  
You can explain why causal LM predicts the next token, why tokenization matters, why `labels` can look identical to `input_ids` in HF’s setup, and why packing text before chunking improves efficiency. The HF forums repeatedly show that these are the exact points where learners get confused. ([Hugging Face Forums](https://discuss.huggingface.co/t/where-does-the-transformers-do-the-target-text-shifting-in-causal-lm/32408 "Where does the Transformers do the target text shifting in ..."))

**Main pitfalls:**

- Confusing **tokenizer training** with **model training**. HF explicitly separates them. ([Hugging Face](https://huggingface.co/learn/llm-course/chapter6/2 "Training a new tokenizer from an old one"))
- Thinking the course example is “real full-scale pretraining.” It is not. HF presents it as a reduced, practical example. ([Hugging Face](https://huggingface.co/learn/llm-course/chapter7/6 "Training a causal language model from scratch"))
- Getting stuck on label shifting. In the common HF causal-LM path, the shift happens inside the model. ([Hugging Face Forums](https://discuss.huggingface.co/t/where-does-the-transformers-do-the-target-text-shifting-in-causal-lm/32408 "Where does the Transformers do the target text shifting in ..."))

* * *

## Track 2 — Build a small LLM yourself

**Use this track when:**  
You want runnable code and a model you actually trained, even if it is small.

**Goal:**  
Train a tiny but real decoder-only model end to end.

**Build in this order:**

1. **`minbpe`**  
Start here for tokenization code. It is a minimal, clean implementation of byte-level BPE, which is commonly used in LLM tokenization. ([GitHub](https://github.com/karpathy/minbpe "karpathy/minbpe: Minimal, clean code for the Byte Pair ..."))

2. **`build-nanogpt`**  
Then build a GPT-like model step by step. The repo is intentionally structured so you can follow the commit history as the system is built gradually. ([GitHub](https://github.com/karpathy/build-nanogpt "Video+code lecture on building nanoGPT from scratch"))

3. **Raschka’s `LLMs-from-scratch`**  
Use this when you want a more complete, careful, beginner-friendly path. The repo explicitly covers developing, pretraining, and finetuning a GPT-like LLM. ([GitHub](https://github.com/rasbt/LLMs-from-scratch "rasbt/LLMs-from-scratch: Implement a ChatGPT-like LLM in ..."))

4. **HF blog: “How to train a new language model from scratch”**  
This is the compact HF version of the whole pipeline: find data, train a tokenizer, train a language model, validate it, then fine-tune it. ([Hugging Face](https://huggingface.co/blog/how-to-train "How to train a new language model from scratch using ..."))

5. **HF blog: “Training CodeParrot from Scratch”**  
Use this when you want a more realistic HF pretraining case than the course toy example. HF describes it as a step-by-step guide to training a large GPT-2 model for code from scratch. ([Hugging Face](https://huggingface.co/blog/codeparrot "Training CodeParrot 🦜 from Scratch"))

6. **LitGPT pretraining tutorial**  
Good modern practical stack after the educational repos. LitGPT documents `litgpt pretrain`, and the project positions itself as high-performance LLM recipes for pretraining, finetuning, and deployment. ([GitHub](https://github.com/Lightning-AI/litgpt/blob/main/tutorials/pretrain.md "litgpt/tutorials/pretrain.md at main"))

7. **TinyStories**  
Best cheap sandbox. The paper shows that very small models can learn coherent English on this dataset, which makes it unusually good for low-budget experiments. ([arXiv](https://arxiv.org/abs/2305.07759 "TinyStories: How Small Can Language Models Be and Still Speak Coherent English?"))

**A good endpoint for Track 2:**  
Train a small model on TinyStories or a narrow code corpus, then inspect generations, training loss, and preprocessing behavior.

**Main pitfalls:**

- Preprocessing can dominate your runtime. A recent HF forum thread shows learners getting stuck on long tokenization and chunking steps in Colab. ([Hugging Face Forums](https://discuss.huggingface.co/t/llm-course-training-a-causal-language-model-from-scratch/173736 "LLM course - training a causal language model from scratch"))
- Packing matters. The efficient pattern is to concatenate samples with EOS and then chunk, instead of wasting short remainders. HF’s course and example scripts use this logic. ([GitHub](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm_no_trainer.py "run\_clm\_no\_trainer.py - huggingface/transformers"))
- Naive perplexity evaluation is misleading for fixed-length models. HF recommends a sliding-window strategy. ([Hugging Face](https://huggingface.co/docs/transformers/perplexity "Perplexity of fixed-length models"))
- Official examples can still break or drift. HF course discussions document typos and deprecated API usage in the training section. ([Hugging Face Forums](https://discuss.huggingface.co/t/chapter-7-questions/11746?page=6 "Chapter 7 questions - Page 6 - Course"))

* * *

## Track 3 — Learn serious pretraining engineering

**Use this track when:**  
You already understand the basics and have trained small models. Now the problem is **throughput, memory, parallelism, stability, and scaling**.

**Goal:**  
Understand how real pretraining systems are organized.

**Read in this order:**

1. **HF LLM Training Playbook**  
Start here for the overview. HF describes it as an open collection of implementation tips, tricks, and resources for training large language models. ([GitHub](https://github.com/huggingface/large_language_model_training_playbook "The Large Language Model Training Playbook"))

2. **HF LLM Training Handbook**  
Then go deeper. HF explicitly says this is technical material for LLM training engineers and operators. ([GitHub](https://github.com/huggingface/llm_training_handbook "huggingface/llm\_training\_handbook"))

3. **Megatron-LM / Megatron Core**  
Study this when you need large-scale distributed training concepts. NVIDIA describes Megatron-LM as a reference example for research teams and distributed training, and Megatron Core as the high-performance building blocks for large-scale generative AI training. ([GitHub](https://github.com/NVIDIA/Megatron-LM "NVIDIA/Megatron-LM: Ongoing research training ..."))

4. **HF Accelerate Megatron-LM guide**  
Useful bridge if you already know HF tooling and want to see how large-scale GPT pretraining connects to their example scripts. ([Hugging Face](https://huggingface.co/docs/accelerate/usage_guides/megatron_lm "Megatron-LM"))

5. **Nanotron**  
Good alternative if you want a simpler, flexible pretraining library that is still designed for speed and scale. ([GitHub](https://github.com/huggingface/nanotron "huggingface/nanotron: Minimalistic large language model ..."))

6. **Pythia**  
Best for studying training dynamics rather than only final checkpoints. The project provides models and many checkpoints specifically to support research into how LMs evolve across training and scale. ([GitHub](https://github.com/EleutherAI/pythia "Pythia: Interpreting Transformers Across Time and Scale"))

**What Track 3 is really about:**  
Not “how to make a toy model run,” but “how to make training stable, efficient, parallel, debuggable, and reproducible.” That is exactly how HF positions its handbook/playbook material. ([GitHub](https://github.com/huggingface/llm_training_handbook "huggingface/llm\_training\_handbook"))

**Main pitfalls:**

- Entering this track too early. These resources assume you already know the basics.
- Treating distributed training as the first thing to learn, instead of the last thing.
- Ignoring real-world friction. Even practical scaling libraries still have active bug reports and setup issues. ([GitHub](https://github.com/huggingface/nanotron/issues "Issues · huggingface/nanotron"))

* * *

## Best default path for most people

For most people, the best order is:

**Track 1 → Track 2 → selective parts of Track 3**

That sequence matches the way the resources themselves are split: the HF course teaches the workflow and a small from-scratch example, while the handbook/playbook and systems libraries target training engineers working on scale and operations. ([Hugging Face](https://huggingface.co/learn/llm-course/chapter7/6 "Training a causal language model from scratch"))

## Short version

- **Track 1** = understand transformers and causal LM training
- **Track 2** = train a small model yourself
- **Track 3** = learn real pretraining systems and scaling

The most common mistake is skipping Track 1, dabbling in Track 3, and never finishing Track 2.