# Finetuning LLama2-70B using 4-bit quantization on multi-GPU using Deepspeed ZeRO

**URL:** <https://discuss.huggingface.co/t/finetuning-llama2-70b-using-4-bit-quantization-on-multi-gpu-using-deepspeed-zero/57528>\
**Category:** Intermediate\
**Created:** [October 5, 2023, 5:21pm UTC](https://discuss.huggingface.co/t/finetuning-llama2-70b-using-4-bit-quantization-on-multi-gpu-using-deepspeed-zero/57528 "2023-10-05T17:21:34Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![prakhar19](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prakhar19/32/20093_2.png) [@prakhar19](https://discuss.huggingface.co/u/prakhar19)\
**Post date:** [October 5, 2023, 5:21pm UTC](https://discuss.huggingface.co/t/finetuning-llama2-70b-using-4-bit-quantization-on-multi-gpu-using-deepspeed-zero/57528/1 "2023-10-05T17:21:35Z")

</div>

Hello,

I am trying to Finetune LLama2-70B 4-bit quantized on multi-GPU (3xA100 40GBs) using Deepspeed ZeRO-3.

At the moment, I am able to Finetune the 4-bit quantized model on the 3 GPUs using SFTTrainer ModelParallel (basically just device\_map: auto).

And following the [DeepSpeed Integration](https://huggingface.co/docs/transformers/main_classes/deepspeed#zero3-example), what I understand is that adding a DeepSpeed config and running the script using deepspeed should have done the trick, but it doesn’t.

It still tries to load the model as ModelParallel (different layers on different GPUs) and also ignores the quantization config, trying to load it completely (which fails).

Can you please help me by directing me to some working examples or maybe something similar?

Thanks a lot in advance

---

<div class="post-metadata">

**Author:** ![asherisaac](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@asherisaac](https://discuss.huggingface.co/u/asherisaac)\
**Post date:** [March 19, 2024, 7:50am UTC](https://discuss.huggingface.co/t/finetuning-llama2-70b-using-4-bit-quantization-on-multi-gpu-using-deepspeed-zero/57528/2 "2024-03-19T07:50:16Z")

</div>

Deepspeed Zero-3 doesn’t work with 4-bit quantization yet. They recently announced support for quantization down to 8-bit, for which you have to make changes to your deepspeed json config file adding a new ‘quantization’ section.