# Deepspeed inference and infinity offload with bitsandbytes 4bit loaded models

**URL:** <https://discuss.huggingface.co/t/deepspeed-inference-and-infinity-offload-with-bitsandbytes-4bit-loaded-models/43929>\
**Category:** DeepSpeed\
**Created:** [June 20, 2023, 1:15pm UTC](https://discuss.huggingface.co/t/deepspeed-inference-and-infinity-offload-with-bitsandbytes-4bit-loaded-models/43929 "2023-06-20T13:15:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![SilkeP](https://avatars.discourse-cdn.com/v4/letter/s/e47774/32.png) [@SilkeP](https://discuss.huggingface.co/u/SilkeP)\
**Post date:** [June 20, 2023, 1:15pm UTC](https://discuss.huggingface.co/t/deepspeed-inference-and-infinity-offload-with-bitsandbytes-4bit-loaded-models/43929/1 "2023-06-20T13:15:22Z")

</div>

Is it possible to use deepspeed inference with a 4/8-bit quantized model using bitsandbytes?

I use the bitsandbytes package like this:

```auto
nf4_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(self.model_id, device_map="auto", quantization_config=self.bnb_config)

zero_config = {
        "stage": 3,
        "offload_param": {
            "device": "cpu"
            }
        }

ds_model = deepspeed.init_inference(
            model=model,     
            mp_size=1,        
            zero=zero_config
        )

 pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=self.max_new_tokens
            )

```

However, it throws an error:

> ValueError: `.to` is not supported for `4-bit` or `8-bit` models. Please use the model as it is, since the model  
> has already been set to the correct devices and casted to the correct `dtype`.

The ultimate goal is to combine the quantization with deepspeed zero infinity offload in the hope to run a larger model that currently does not fit on my GPU.

---

<div class="post-metadata">

**Author:** ![Johnguitarist](https://avatars.discourse-cdn.com/v4/letter/j/67e7ee/32.png) [@Johnguitarist](https://discuss.huggingface.co/u/Johnguitarist)\
**Post date:** [June 30, 2023, 4:58pm UTC](https://discuss.huggingface.co/t/deepspeed-inference-and-infinity-offload-with-bitsandbytes-4bit-loaded-models/43929/2 "2023-06-30T16:58:46Z")

</div>

Hey,  
Also currently trying to achieve the same.

Did you manage to get bitsandbytes 4bit and deepspeed working together?

My error is slightly different:  
ValueError: `.half()` is not supported for `4-bit` or `8-bit` models. Please use the model as it is, since the model has already been casted to the correct `dtype`.

Consider this a bump 🙂

---

<div class="post-metadata">

**Author:** ![AvivB](https://avatars.discourse-cdn.com/v4/letter/a/71c47a/32.png) [@AvivB](https://discuss.huggingface.co/u/AvivB)\
**Post date:** [July 27, 2023, 8:46pm UTC](https://discuss.huggingface.co/t/deepspeed-inference-and-infinity-offload-with-bitsandbytes-4bit-loaded-models/43929/3 "2023-07-27T20:46:32Z")

</div>

I am getting this error too.

Edit: There appears to be support now:

> **[DeepSpeed Mixture-of-Quantization (MoQ)](https://www.deepspeed.ai/tutorials/MoQ-tutorial/#deepspeed-configuration-file)**
>
> DeepSpeed introduces new support for model compression using quantization, called Mixture-of-Quantization (MoQ). MoQ is designed on top of QAT (Quantization-Aware Training), with the difference that it schedules various data precisions across the...