# Train from scratch: models and efficiency with 1 GPU

**URL:** <https://discuss.huggingface.co/t/train-from-scratch-models-and-efficiency-with-1-gpu/10243>\
**Category:** Models\
**Created:** [September 23, 2021, 11:00am UTC](https://discuss.huggingface.co/t/train-from-scratch-models-and-efficiency-with-1-gpu/10243 "2021-09-23T11:00:20Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mattivi](https://avatars.discourse-cdn.com/v4/letter/m/e9a140/32.png) [@mattivi](https://discuss.huggingface.co/u/mattivi)\
**Post date:** [September 23, 2021, 11:00am UTC](https://discuss.huggingface.co/t/train-from-scratch-models-and-efficiency-with-1-gpu/10243/1 "2021-09-23T11:00:20Z")

</div>

Hi,  
I need to train from scratch a Transformer model and the limitation is to have only 1 GPU.

- Which Transformer model is the most efficient one to train on 1 GPU, with a good trade-off between accuracy and training speed?  
@sgugger is there available a comparison about training time Vs. BLEU accuracy metrics for different models that are supported in Huggingface library (train from scratch)?

For example:

- ELECTRA-base ([paper here](https://arxiv.org/pdf/2003.10555.pdf)) is shown at page 2 to reach higher BLEU score compared to BERT-base keeping same Pre-train FLOPs. Or, from another point of view, one can create an ELECTRA model (somewhere between ELECTRA-small and ELECTRA-base) that can reach the same BLEU score as BERT-base, but with less Pre-train FLOPs. In this manner, the pre-training time and resources is optimized. However, ELECTRA train from scratch is not supported in Huggingface library.
- FNet ([paper here](https://arxiv.org/pdf/2105.03824.pdf)) is shown to achieve 92% of BERT-Base accuracy on the GLUE benchmark, but with a train that is 80% faster on GPU (page 9).

Any suggestion which model to choose and which best configuration (e.g. DeepSpeed as well) to achieve a BERT-base BLEU score but with much less training time on 1 GPU?  
Thanks a lot!  
Riccardo