# MFU number with Finetuning llama 70b using fsdp

**URL:** <https://discuss.huggingface.co/t/mfu-number-with-finetuning-llama-70b-using-fsdp/58232>\
**Category:** 🤗Transformers\
**Created:** [October 11, 2023, 11:54pm UTC](https://discuss.huggingface.co/t/mfu-number-with-finetuning-llama-70b-using-fsdp/58232 "2023-10-11T23:54:05Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![xwjiang2010](https://avatars.discourse-cdn.com/v4/letter/x/a88e57/32.png) [@xwjiang2010](https://discuss.huggingface.co/u/xwjiang2010)\
**Post date:** [October 11, 2023, 11:54pm UTC](https://discuss.huggingface.co/t/mfu-number-with-finetuning-llama-70b-using-fsdp/58232/1 "2023-10-11T23:54:05Z")

</div>

I have a question regarding this [blogpost](https://huggingface.co/blog/ram-efficient-pytorch-fsdp). In the calculation of MFU, on bullet point 4, should we divide by 16 A100’s worth of peak FLOPs? This brings MFU to a mere 2.8%. Is this expected or my understanding is off?