# Finetuning and single-GPU utilization

**URL:** <https://discuss.huggingface.co/t/finetuning-and-single-gpu-utilization/9353>\
**Category:** 🤗Transformers\
**Created:** [August 19, 2021, 2:10pm UTC](https://discuss.huggingface.co/t/finetuning-and-single-gpu-utilization/9353 "2021-08-19T14:10:16Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![shahjaidev](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shahjaidev/32/2531_2.png) [@shahjaidev](https://discuss.huggingface.co/u/shahjaidev)\
**Post date:** [August 19, 2021, 2:10pm UTC](https://discuss.huggingface.co/t/finetuning-and-single-gpu-utilization/9353/1 "2021-08-19T14:10:16Z")

</div>

Hi all!  
I have a question related to fine-tuning:  
Is fine-tuning a pre-trained transformer model a easier model an ‘easier’ task than training a transformer from scratch (BERT, GPT-2) in terms of GPU needs and GPU memory usage?

To clarify further, I’ve read how to train most transformer models, one would require multi-GPU training. However, is it possible to fine-tune some of these models on a single-GPU?  
Why is this the case? Is it because the computational graph during backpropagation is smaller when a lot of the parameters are frozen?