# I reproduced TinyStories using Llama style architecture from scratch. Feedback required

**URL:** <https://discuss.huggingface.co/t/i-reproduced-tinystories-using-llama-style-architecture-from-scratch-feedback-required/176772>\
**Category:** Models\
**Created:** [June 14, 2026, 1:59am UTC](https://discuss.huggingface.co/t/i-reproduced-tinystories-using-llama-style-architecture-from-scratch-feedback-required/176772 "2026-06-14T01:59:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![vinothai](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vinothai/32/60789_2.png) [@vinothai](https://discuss.huggingface.co/u/vinothai)\
**Post date:** [June 14, 2026, 1:59am UTC](https://discuss.huggingface.co/t/i-reproduced-tinystories-using-llama-style-architecture-from-scratch-feedback-required/176772/1 "2026-06-14T01:59:53Z")

</div>

Hello everyone,

Model Name: Llama-TinyStories

Model Link: [vinothkumarn/Llama-TinyStories · Hugging Face](https://huggingface.co/vinothkumarn/Llama-TinyStories)

This is my first attempt at training and releasing a model. It was a fun TinyStories reproduction experiment and I learnt a lot doing it. I did a detailed writeup about the training process and mentioned full configuration details in this post,

> **[From Gibberish to Stories: Reproducing TinyStories using LLaMA architecture](https://mlexperiments.substack.com/p/from-gibberish-to-stories-reproducing)**
>
> Detailed walkthrough of reproducing TinyStories using Llama architecture

Can I get some feedback about the configuration? Any suggestions for reducing the hallucinations?

I did 2.08 epochs of TinyStories dataset. Original model did 20 epochs. What is the optimal epoch size and how do we determine that?

Thank you!

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 14, 2026, 5:33am UTC](https://discuss.huggingface.co/t/i-reproduced-tinystories-using-llama-style-architecture-from-scratch-feedback-required/176772/2 "2026-06-14T05:33:00Z")

</div>

There seem to be many useful follow-up lessons around TinyStories since the original paper, especially around evaluation and small-model training. But for this case, I think the most practical next step may be to establish a small evaluation loop:

* * *

First, nice work. Training and releasing a from-scratch TinyStories-style model is already a useful experiment, especially with a public write-up and reproducible-looking training notes.

My main suggestion would be:

> Freeze the evaluation recipe before changing the training recipe.

I would not start by asking “what is the optimal epoch count?” directly. I would reframe it as:

> Which checkpoint gives the best TinyStories behavior under the same prompts, same decoding settings, same seed list, and same scoring rubric?

The original `20 epochs` is a useful reference point, but I would not treat it as a magic target. Your setup differs from the original [TinyStories](https://arxiv.org/abs/2305.07759) setup in architecture, batch/effective batch, learning rate, tokenizer, training framework, and probably generation settings. So I would choose the checkpoint empirically rather than choosing the epoch count directly.

## 1. Convert the question into a checkpoint comparison

I would make a small table like this and fill it for several checkpoint candidates.

| checkpoint | epoch\_equiv | tokens\_seen | val\_loss | ppl | grammar | creativity | consistency | invented\_prior\_event\_rate | repetition\_rate | eos\_rate | max\_new\_tokens\_hit\_rate | short verdict |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| current | 2.08 | ~982M | … | … | … | … | … | … | … | … | … | current release |
| ckpt\_4e | 4.00 | … | … | … | … | … | … | … | … | … | … | compare |
| ckpt\_8e | 8.00 | … | … | … | … | … | … | … | … | … | … | compare |
| ckpt\_12e | 12.00 | … | … | … | … | … | … | … | … | … | … | compare |
| ckpt\_20e | 20.00 | … | … | … | … | … | … | … | … | … | … | original-like exposure |

This turns:

> “Should I train 20 epochs?”

into:

> “Does the 20-epoch checkpoint actually improve consistency and stopping behavior, or does it only improve surface fluency, loss, or memorized TinyStories patterns?”

The best checkpoint may not be the final checkpoint. More epochs should be treated as a hypothesis to test.

## 2. Split “hallucination” into measurable failure modes

I understand what you mean by hallucination, but for TinyStories I would make the term more specific.

In fiction generation, adding new events is normal. That is the task. I would only count it as a failure when the model contradicts the established story state, invents a prior event, mixes up who said/knew/did what, loops, or fails to stop naturally.

For your soup example, I would label the issue as something like:

- `invented_prior_event`
- `dialogue_memory_error`
- `story_state_tracking_failure`
- `narrative_inconsistency`

rather than factual hallucination in the usual QA/RAG sense.

A practical failure taxonomy could be:

| broad concern | TinyStories-specific label | concrete test |
| --- | --- | --- |
| hallucination | `invented_prior_event` | Does the model rely on an event not established earlier? |
| hallucination | `contradicted_prompt` | Does the continuation violate the prompt/story state? |
| hallucination | `entity_confusion` | Does it mix up Tom/Jane/Lily/etc.? |
| hallucination | `dialogue_memory_error` | Does it forget who said what? |
| hallucination | `knowledge_state_error` | Does a character know something they should not know? |
| weird output | `repetition_issue` | Does it loop or repeat phrases/sentences? |
| weird output | `bad_stopping` | Does it stop naturally or only because of the token limit? |
| normal creativity | compatible new event | Do not penalize if it fits the established story state |

This fits the original TinyStories evaluation idea pretty well. The [TinyStories paper](https://arxiv.org/abs/2305.07759) uses GPT-4-style grading for dimensions like grammar, creativity, and consistency. In this case, the consistency axis is probably the important one.

## 3. Build a small evaluation artifact

I would make the evaluation concrete. Something like:

```plaintext
prompts.jsonl
generation_config.json
checkpoints.txt
outputs.jsonl
scores.csv

```

The goal is not to build a huge benchmark. The goal is to make the next training decision less ambiguous.

Minimum viable setup:

| artifact | minimum version | better version |
| --- | --- | --- |
| `prompts.jsonl` | 20 official + 10 stress prompts | official prompts + tagged stress prompts |
| `generation_config.json` | one fixed decoding profile | stable profile + story-sampling profile |
| `checkpoints.txt` | current + 2 future checkpoints | 2 / 4 / 8 / 12 / 20 epoch-equivalent |
| `outputs.jsonl` | prompt, seed, completion | plus length, stop reason, decoding profile |
| `scores.csv` | manual notes | JSON judge + manual audit + mechanical metrics |

## 4. Use three prompt groups

I would not evaluate only on a few nice-looking samples.

Use three prompt groups:

| prompt group | purpose | source |
| --- | --- | --- |
| validation split | training signal | `tinystories-valid.txt` |
| official eval prompts | comparability with TinyStories-style evaluation | `Evaluation_prompts.yaml` |
| custom stress prompts | detect story-state failures | hand-written small set |

The [TinyStories dataset README](https://huggingface.co/datasets/roneneldan/TinyStories/blob/main/README.md) mentions `tinystories-valid.txt`, `Evaluation_prompts.yaml`, and `TinyStoriesV2-GPT4-train.txt`, so those are natural starting points.

For the custom stress prompts, I would deliberately test the exact failure modes you care about.

Example stress prompts:

```plaintext
Tom put the red cup on the table. Jane did not see him move it. Later, Jane came into the room and looked for the cup.

```

Tests:

- object-state tracking
- knowledge-state tracking
- whether Jane “knows” something she should not know

```plaintext
Mia wanted to warn Ben about the blue box, but she did not tell him yet. Ben walked into the room and smiled.

```

Tests:

- invented prior event
- knowledge-state error
- dialogue memory

```plaintext
The soup tasted bitter, but Tom had not said anything about it. Jane took a small sip and made a face.

```

Tests:

- the same kind of issue as your soup example
- whether the model invents “I told you it was bitter”

```plaintext
Lily told Sam, “The kitten is under the chair.” Sam nodded. Then Lily left the room, and the kitten crawled into the basket.

```

Tests:

- character memory
- object location update
- whether the model preserves old vs new state

Keep the language simple. The point is not to make hard benchmark tasks. The point is to create TinyStories-shaped prompts that expose consistency failures.

## 5. Freeze decoding settings

For checkpoint comparison, I would save one fixed [GenerationConfig](https://huggingface.co/docs/transformers/en/main_classes/text_generation) and reuse it unchanged across checkpoints.

I would prefer `max_new_tokens` over `max_length`, because `max_new_tokens` controls only newly generated tokens, while `max_length` includes the prompt length. That matters when prompt lengths differ.

Example evaluation config:

```json
{
  "max_new_tokens": 192,
  "do_sample": true,
  "temperature": 0.7,
  "top_p": 0.9,
  "repetition_penalty": 1.05,
  "eos_token_id": "<eos_token_id>",
  "pad_token_id": "<eos_token_id>"
}

```

The exact numbers are not the key point. The key point is that the same settings are used across all checkpoints.

I would avoid tuning decoding separately per checkpoint. Otherwise it becomes hard to tell whether the improvement came from training or decoding.

## 6. Use a seed list, not one sample

A single sample can be misleading. I would use a fixed seed list.

| evaluation mode | seeds per prompt |
| --- | --- |
| quick smoke test | 1 |
| checkpoint comparison | 3 |
| stronger sample quality eval | 5 |
| TinyStories-style GPT-Eval-ish setup | 10 |

So a small but useful evaluation could be:

| item | value |
| --- | --- |
| official prompts | 20 |
| stress prompts | 10 |
| total prompts | 30 |
| seeds per prompt | 3 |
| checkpoints | 3 |
| total generations | 270 |

That is not huge, but it is already much more informative than a few cherry-picked generations.

## 7. Log sample-level outputs

For each generation, I would log one row.

Example `outputs.jsonl` row:

```json
{
  "checkpoint": "ckpt_4e",
  "epoch_equiv": 4.0,
  "tokens_seen": 1887490068,
  "prompt_id": "stress_002",
  "prompt_source": "custom",
  "prompt_tags": ["dialogue_memory", "invented_prior_event"],
  "seed": 1,
  "decoding_profile": "eval_story",
  "prompt": "The soup tasted bitter, but Tom had not said anything about it. Jane took a small sip and made a face.",
  "completion": "...",
  "length_tokens": 142,
  "stop_reason": "eos"
}

```

Example `scores.csv` / `scores.jsonl` row:

```json
{
  "checkpoint": "ckpt_4e",
  "prompt_id": "stress_002",
  "seed": 1,
  "grammar": 4,
  "creativity": 3,
  "consistency": 2,
  "invented_prior_event": true,
  "entity_confusion": false,
  "dialogue_memory_error": true,
  "repetition_issue": false,
  "bad_stopping": false,
  "human_note": "Invented a prior warning by Tom."
}

```

Then you can summarize failures like:

| metric | value |
| --- | --- |
| `invented_prior_event_rate` | 12 / 150 |
| `entity_confusion_rate` | 9 / 150 |
| `dialogue_memory_error_rate` | 14 / 150 |
| `repetition_issue_rate` | 18 / 150 |
| `max_new_tokens_hit_rate` | 20 / 150 |

This turns “the model hallucinates sometimes” into measurable buckets.

If this becomes useful later, it can be formalized with [Lighteval](https://huggingface.co/docs/lighteval/en/index), a [Lighteval custom task](https://huggingface.co/docs/lighteval/adding-a-custom-task), or [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness). But I would start with a simple script.

## 8. Use a JSON judge, plus manual audit

A TinyStories-style judge prompt could be:

```plaintext
You are evaluating a short children's story continuation.

Prompt:
<prompt>

Completion:
<completion>

Score from 1 to 5:
- grammar: Is the English grammatical and age-appropriate?
- creativity: Is the continuation interesting without drifting too far?
- consistency: Does it preserve the established story state?
- entity_tracking: Does it keep track of who did/knew/said what?
- ending: Does it stop naturally?

Also return booleans:
- invented_prior_event
- entity_confusion
- dialogue_memory_error
- repetition_or_loop
- contradicted_prompt
- bad_stopping

Return only JSON.

```

Example output:

```json
{
  "grammar": 4,
  "creativity": 3,
  "consistency": 2,
  "entity_tracking": 2,
  "ending": 3,
  "invented_prior_event": true,
  "entity_confusion": false,
  "dialogue_memory_error": true,
  "repetition_or_loop": false,
  "contradicted_prompt": true,
  "bad_stopping": false,
  "short_reason": "The continuation invents a prior warning that was not established in the prompt."
}

```

I would use this as a diagnostic tool, not as absolute truth. A small manual audit of 20–30 samples is still useful, especially for consistency errors.

## 9. Measure repetition and stopping mechanically

Do not make the judge do everything.

Some metrics are easier to compute mechanically:

| metric | meaning |
| --- | --- |
| `length_tokens` | generated token length |
| `ended_with_eos` | whether generation stopped with EOS |
| `hit_max_new_tokens` | whether generation ran to the limit |
| `unique_token_ratio` | rough diversity signal |
| `repeated_3gram_ratio` | local repetition |
| `repeated_sentence_count` | sentence-level loops |
| `unfinished_last_sentence` | possibly bad stopping |

This helps separate three different problems:

| symptom | likely bucket |
| --- | --- |
| model invents a previous warning | story-state consistency |
| model repeats the same moral sentence | degeneration / decoding |
| model never stops until token limit | stopping / EOS / decoding |
| model is grammatical but contradicts the prompt | consistency |
| model is creative but still compatible | not necessarily a failure |

## 10. Configuration notes: what I would check before architecture

I would not jump directly to architecture changes. I would first make the evaluation stable enough to tell what changed.

| priority | check | why |
| --- | --- | --- |
| 1 | validation loss trajectory | tells whether training is still improving |
| 2 | fixed generation settings | separates training behavior from decoding behavior |
| 3 | checkpoint sample table | shows whether consistency improves with exposure |
| 4 | repetition / EOS stats | separates hallucination-like issues from degeneration |
| 5 | LR / effective batch | these differ from the original setup |
| 6 | data hygiene | small models are sensitive to artifacts and repetition |
| 7 | tokenizer ablation | interesting for a second run |
| 8 | architecture changes | useful, but I would do this after eval is clear |

For data hygiene, I would check:

| check | example |
| --- | --- |
| encoding artifacts | `â€`, `Ã`, `�` |
| exact duplicates | story hash |
| near duplicates | fuzzy/minhash check |
| repeated boilerplate | “Once upon a time” frequency |
| length distribution | extremely short/long stories |
| train/valid leakage | exact overlap |
| data variant | original vs GPT-4-only TinyStoriesV2 |

This is not because I think the dataset is bad. It is because small from-scratch models can be sensitive to small data artifacts.

## 11. TinyStories hindsight: useful, but narrow

One useful hindsight point is that TinyStories is a great controlled lab, but not a complete proxy for general small-language-model behavior.

Later work such as [Readability ≠ Learnability](https://openreview.net/forum?id=AFMGbq39bQ) suggests that learnability may come partly from statistical simplicity and predictable patterns, not only from child-readable language. [SimpleStories](https://arxiv.org/abs/2504.09184) also discusses limitations of TinyStories-style data and proposes a more parameterized synthetic story setup.

So I would use the official TinyStories assets for comparability, but I would not rely only on fluent-looking samples from easy story openings. A model can be fluent on TinyStories-like text while still failing at state tracking.

## 12. My practical recommendation

I would run a small experiment like this:

| step | action |
| --- | --- |
| 1 | create `prompts.jsonl` with official + stress prompts |
| 2 | create one fixed `generation_config.json` |
| 3 | choose 3–5 seeds |
| 4 | generate from current, 4e, 8e, 12e, 20e checkpoints |
| 5 | compute validation loss / perplexity |
| 6 | score grammar / creativity / consistency |
| 7 | label invented prior events, entity confusion, repetition, bad stopping |
| 8 | summarize by checkpoint |
| 9 | choose the checkpoint with the best balance, not just the lowest loss |

The final decision rule could be:

| observation | likely next move |
| --- | --- |
| val loss and consistency both improve | continue training may help |
| val loss improves but consistency does not | add stress eval / inspect data / maybe capacity issue |
| repetition increases | inspect decoding, data formulaicity, overtraining |
| many outputs hit `max_new_tokens` | inspect EOS / stopping / generation config |
| only official prompts look good | add harder stress prompts |
| stress prompts improve with epochs | more training may be justified |
| stress prompts stay bad | consider LR/batch/data/tokenizer/architecture ablations |

So my short version would be:

> Freeze prompts, decoding, seeds, and scoring first. Then compare checkpoints. After that, decide whether the fix is more epochs, lower LR, larger effective batch, different decoding, data cleanup, tokenizer ablation, or architecture changes.

That would make the experiment much easier to reason about.

---

<div class="post-metadata">

**Author:** ![vinothai](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/vinothai/32/60789_2.png) [@vinothai](https://discuss.huggingface.co/u/vinothai)\
**Post date:** [June 17, 2026, 7:31am UTC](https://discuss.huggingface.co/t/i-reproduced-tinystories-using-llama-style-architecture-from-scratch-feedback-required/176772/3 "2026-06-17T07:31:06Z")

</div>

Thanks a lot for the valuable info. I will try it out 👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [June 17, 2026, 7:31pm UTC](https://discuss.huggingface.co/t/i-reproduced-tinystories-using-llama-style-architecture-from-scratch-feedback-required/176772/4 "2026-06-17T19:31:59Z")

</div>

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.