# OOM when I using torch.nn.parallel.DistributedDataParallel to train LLAMA-7B

**URL:** <https://discuss.huggingface.co/t/oom-when-i-using-torch-nn-parallel-distributeddataparallel-to-train-llama-7b/39587>\
**Category:** Beginners\
**Created:** [May 12, 2023, 12:29am UTC](https://discuss.huggingface.co/t/oom-when-i-using-torch-nn-parallel-distributeddataparallel-to-train-llama-7b/39587 "2023-05-12T00:29:34Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![daisr](https://avatars.discourse-cdn.com/v4/letter/d/a9a28c/32.png) [@daisr](https://discuss.huggingface.co/u/daisr)\
**Post date:** [May 12, 2023, 12:29am UTC](https://discuss.huggingface.co/t/oom-when-i-using-torch-nn-parallel-distributeddataparallel-to-train-llama-7b/39587/1 "2023-05-12T00:29:34Z")

</div>

I intended to use 4 NVIDIA 3090 GPUs to train LLAMA-7B(float16) with DistributedDataParallel ,but OOM occured. The llama-7b occupied around 15G memory in one gpu(24G in total), but once I called the function torch.nn.parallel.DistributedDataParallel(), the error happens.

-----the code is like below---------  
model=LlamaForCausalLM.from\_pretrained(“xxxx”,torch\_dtype=torch.float16)  
model.to(device) #running ok here ,GPU memory is 15G/24G  
model = torch.nn.parallel.DistributedDataParallel(model, device\_ids=[args.local\_rank], output\_device=args.local\_rank, find\_unused\_parameters=True) #Out of memory  
happened here

I would like to know why this happens after calling the DDP function and why memory is reallocated again? Does anyone know? I’m very grateful for that.