# Total\_flos vs C = 6 \* N \* D

**URL:** <https://discuss.huggingface.co/t/total-flos-vs-c-6-n-d/65593>\
**Category:** 🤗Transformers\
**Created:** [December 13, 2023, 12:02am UTC](https://discuss.huggingface.co/t/total-flos-vs-c-6-n-d/65593 "2023-12-13T00:02:37Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ricdomolm](https://avatars.discourse-cdn.com/v4/letter/r/a9a28c/32.png) [@ricdomolm](https://discuss.huggingface.co/u/ricdomolm)\
**Post date:** [December 13, 2023, 12:02am UTC](https://discuss.huggingface.co/t/total-flos-vs-c-6-n-d/65593/1 "2023-12-13T00:02:37Z")

</div>

As far as I am aware, `total_flos`, returned by `trainer.train`, is supposed to indicate the total number of floating point operations performed to train some given model. However, this value can vary substantially compared to that of the commonly used formula C \approx 6 \* N \* D, where N is number of model parameters and D is model size.

For instance, as a quick experiment, training a freshly initialized 106M parameter Llama model on 52M tokens results in total\_flos=2.7e16, whereas using the formula we obtain C=3.3e16 FLOPS.

My question is whether the returned `total_flos` is indeed accurate (specially for Llama-type models).

Tagging @teven since I believe he wrote the relevant code.

---

<div class="post-metadata">

**Author:** ![ricdomolm](https://avatars.discourse-cdn.com/v4/letter/r/a9a28c/32.png) [@ricdomolm](https://discuss.huggingface.co/u/ricdomolm)\
**Post date:** [December 13, 2023, 12:31pm UTC](https://discuss.huggingface.co/t/total-flos-vs-c-6-n-d/65593/2 "2023-12-13T12:31:38Z")

</div>

Currently, trainer does not include the embedding parameters in the FLOP computation, which is the source of the discrepancy observed 🙂