# Huge disparity between CPU and GPU memory usage?

**URL:** <https://discuss.huggingface.co/t/huge-disparity-between-cpu-and-gpu-memory-usage/15022>\
**Category:** 🤗Transformers\
**Created:** [February 22, 2022, 9:24am UTC](https://discuss.huggingface.co/t/huge-disparity-between-cpu-and-gpu-memory-usage/15022 "2022-02-22T09:24:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jrandel](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jrandel](https://discuss.huggingface.co/u/jrandel)\
**Post date:** [February 22, 2022, 9:24am UTC](https://discuss.huggingface.co/t/huge-disparity-between-cpu-and-gpu-memory-usage/15022/1 "2022-02-22T09:24:57Z")

</div>

So I’ve got this T5-large training pipeline going, on a CPU-only VM with 16GB of total memory, and with optimization of both gradient\_accumulation\_steps and using gradient\_checkpointing = True, I can get training started with a train batch size of 32 and eval batch size of 32.

However on Colab Pro+ with 55GB of system memory and a V100 GPU with 16GB and using the same hyperparams, I can’t even get above a batch size of 2…?

Any thoughts?