# Deepspeed inference stage 3 + quantization

**URL:** <https://discuss.huggingface.co/t/deepspeed-inference-stage-3-quantization/76549>\
**Category:** DeepSpeed\
**Created:** [March 8, 2024, 3:25pm UTC](https://discuss.huggingface.co/t/deepspeed-inference-stage-3-quantization/76549 "2024-03-08T15:25:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![SilkeP33](https://avatars.discourse-cdn.com/v4/letter/s/ba9def/32.png) [@SilkeP33](https://discuss.huggingface.co/u/SilkeP33)\
**Post date:** [March 8, 2024, 3:25pm UTC](https://discuss.huggingface.co/t/deepspeed-inference-stage-3-quantization/76549/1 "2024-03-08T15:25:19Z")

</div>

trying to set deepspeed with inference zero 3 as follows:

```auto
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch 
from transformers.integrations import HfDeepSpeedConfig

with torch.no_grad():
    hfds_config = HfDeepSpeedConfig(config_file_or_dict="config.json")
    # Now model is on-the-fly quantized.
    tokenizer = AutoTokenizer.from_pretrained("google/gemma-2b-it")
    model = AutoModelForCausalLM.from_pretrained("google/gemma-2b-it")

input_text = "Query"
input_ids = tokenizer(input_text, return_tensors="pt")

outputs = model.generate(**input_ids, max_length= 200)
print(tokenizer.decode(outputs[0]))

```

With config.json the deepspeed config file as follows:

```auto
{
    "weight_quantization": {
        "quantized_initialization": {
            "num_bits": 4,
            "group_size": 64,
            "group_dim": 1,
            "symmetric": false
        }
    }, 
    "zero_optimization": {
          "stage": 3,
          "offload_optimizer": {
              "device": "cpu",
              "pin_memory": true
          },
          "offload_param": {
              "device": "cpu",
              "pin_memory": true
          },
          "overlap_comm": true,
          "contiguous_gradients": true,
          "sub_group_size": 1e9,
          "reduce_bucket_size": "auto",
          "stage3_prefetch_bucket_size": "auto",
          "stage3_param_persistence_threshold": "auto",
          "stage3_max_live_parameters": 1e9,
          "stage3_max_reuse_distance": 1e9,
          "stage3_gather_fp16_weights_on_model_save": true
      }, 
      "train_batch_size": 32, 
      "torch_dtype": "float32"
  }

```

and the error as follows:

> [2024-03-08 15:12:19,262] [INFO] [real\_accelerator.py:191:get\_accelerator] Setting ds\_accelerator to cuda (auto detect)  
> [2024-03-08 15:12:19,600] [WARNING] [config\_utils.py:69:\_process\_deprecated\_field] Config parameter stage3\_gather\_fp16\_weights\_on\_model\_save is deprecated use gather\_16bit\_weights\_on\_model\_save instead  
> [2024-03-08 15:12:19,601] [INFO] [comm.py:637:init\_distributed] cdb=None  
> [2024-03-08 15:12:19,602] [INFO] [comm.py:652:init\_distributed] Not using the DeepSpeed or dist launchers, attempting to detect MPI environment…  
> [2024-03-08 15:12:19,624] [INFO] [comm.py:702:mpi\_discovery] Discovered MPI settings of world\_rank=0, local\_rank=0, world\_size=1, master\_addr=10.0.0.4, master\_port=29500  
> [2024-03-08 15:12:19,626] [INFO] [comm.py:668:init\_distributed] Initializing TorchBackend in DeepSpeed with backend nccl  
> [2024-03-08 15:12:19,635] [INFO] [partition\_parameters.py:559:patch\_init\_and\_builtins] Enable Zero3 engine with INT4 quantization.  
> [2024-03-08 15:12:22,156] [INFO] [partition\_parameters.py:343: **exit**] finished initializing model - num\_params = 165, num\_elems = 3.03B  
> {  
> “name”: “RuntimeError”,  
> “message”: “self.size(-1) must be divisible by 2 to view BFloat16 as Float (different element sizes), but got 1”,  
> “stack”: "---------------------------------------------------------------------------  
> RuntimeError Traceback (most recent call last)  
> Cell In[1], line 10  
> 8 # Now model is on-the-fly quantized.  
> 9 tokenizer = AutoTokenizer.from\_pretrained("google/gemma-2b-it")  
> —\> 10 model = AutoModelForCausalLM.from\_pretrained("google/gemma-2b-it")  
> 12 input\_text = "Query"  
> 13 input\_ids = tokenizer(input\_text, return\_tensors="pt")
> 
> File ~/.local/lib/python3.8/site-packages/transformers/models/auto/auto\_factory.py:561, in \_BaseAutoModelClass.from\_pretrained(cls, pretrained\_model\_name\_or\_path, \*model\_args, \*\*kwargs)  
> 559 elif type(config) in cls.\_model\_mapping.keys():  
> 560 model\_class = \_get\_model\_class(config, cls.\_model\_mapping)  
> → 561 return model\_class.from\_pretrained(  
> 562 pretrained\_model\_name\_or\_path, \*model\_args, config=config, \*\*hub\_kwargs, \*\*kwargs  
> 563 )  
> 564 raise ValueError(  
> 565 f"Unrecognized configuration class {config. **class** } for this kind of AutoModel: {cls. **name** }.  
> "  
> 566 f"Model type should be one of {', '.join(c. **name** for c in cls.\_model\_mapping.keys())}."  
> 567 )
> 
> File ~/.local/lib/python3.8/site-packages/transformers/modeling\_utils.py:3502, in PreTrainedModel.from\_pretrained(cls, pretrained\_model\_name\_or\_path, config, cache\_dir, ignore\_mismatched\_sizes, force\_download, local\_files\_only, token, revision, use\_safetensors, \*model\_args, \*\*kwargs)  
> 3493 if dtype\_orig is not None:  
> 3494 torch.set\_default\_dtype(dtype\_orig)  
> 3495 (  
> 3496 model,  
> 3497 missing\_keys,  
> 3498 unexpected\_keys,  
> 3499 mismatched\_keys,  
> 3500 offload\_index,  
> 3501 error\_msgs,  
> → 3502 ) = cls.\_load\_pretrained\_model(  
> 3503 model,  
> 3504 state\_dict,  
> 3505 loaded\_state\_dict\_keys, # XXX: rename?  
> 3506 resolved\_archive\_file,  
> 3507 pretrained\_model\_name\_or\_path,  
> 3508 ignore\_mismatched\_sizes=ignore\_mismatched\_sizes,  
> 3509 sharded\_metadata=sharded\_metadata,  
> 3510 \_fast\_init=\_fast\_init,  
> 3511 low\_cpu\_mem\_usage=low\_cpu\_mem\_usage,  
> 3512 device\_map=device\_map,  
> 3513 offload\_folder=offload\_folder,  
> 3514 offload\_state\_dict=offload\_state\_dict,  
> 3515 dtype=torch\_dtype,  
> 3516 hf\_quantizer=hf\_quantizer,  
> 3517 keep\_in\_fp32\_modules=keep\_in\_fp32\_modules,  
> 3518 )  
> 3520 # make sure token embedding weights are still tied if needed  
> 3521 model.tie\_weights()
> 
> File ~/.local/lib/python3.8/site-packages/transformers/modeling\_utils.py:3945, in PreTrainedModel.\_load\_pretrained\_model(cls, model, state\_dict, loaded\_keys, resolved\_archive\_file, pretrained\_model\_name\_or\_path, ignore\_mismatched\_sizes, sharded\_metadata, \_fast\_init, low\_cpu\_mem\_usage, device\_map, offload\_folder, offload\_state\_dict, dtype, hf\_quantizer, keep\_in\_fp32\_modules)  
> 3943 error\_msgs += new\_error\_msgs  
> 3944 else:  
> → 3945 error\_msgs += \_load\_state\_dict\_into\_model(model\_to\_load, state\_dict, start\_prefix)  
> 3947 # force memory release  
> 3948 del state\_dict
> 
> File ~/.local/lib/python3.8/site-packages/transformers/modeling\_utils.py:626, in \_load\_state\_dict\_into\_model(model\_to\_load, state\_dict, start\_prefix)  
> 623 if child is not None:  
> 624 load(child, state\_dict, prefix + name + ".")  
> → 626 load(model\_to\_load, state\_dict, prefix=start\_prefix)  
> 627 # Delete `state_dict` so it could be collected by GC earlier. Note that `state_dict` is a copy of the argument, so  
> 628 # it’s safe to delete it.  
> 629 del state\_dict
> 
> File ~/.local/lib/python3.8/site-packages/transformers/modeling\_utils.py:624, in \_load\_state\_dict\_into\_model..load(module, state\_dict, prefix)  
> 622 for name, child in module.\_modules.items():  
> 623 if child is not None:  
> → 624 load(child, state\_dict, prefix + name + ".")
> 
> File ~/.local/lib/python3.8/site-packages/transformers/modeling\_utils.py:624, in \_load\_state\_dict\_into\_model..load(module, state\_dict, prefix)  
> 622 for name, child in module.\_modules.items():  
> 623 if child is not None:  
> → 624 load(child, state\_dict, prefix + name + ".")
> 
> File ~/.local/lib/python3.8/site-packages/transformers/modeling\_utils.py:618, in \_load\_state\_dict\_into\_model..load(module, state\_dict, prefix)  
> 616 with deepspeed.zero.GatheredParameters(params\_to\_gather, modifier\_rank=0):  
> 617 if torch.distributed.get\_rank() == 0:  
> → 618 module.\_load\_from\_state\_dict(\*args)  
> 619 else:  
> 620 module.\_load\_from\_state\_dict(\*args)
> 
> File ~/.local/lib/python3.8/site-packages/deepspeed/inference/quantization/utils.py:269, in wrap\_load\_from\_state\_dict..wrapper(model, state\_dict, prefix, local\_metadata, strict, missing\_keys, unexpected\_keys, error\_msgs)  
> 267 quantized\_weight, quant\_scale, quant\_min = model.weight.quantizer.quantize(state\_dict[key])  
> 268 quantized\_weight = quantized\_weight.view(model.weight.dtype)  
> → 269 quant\_scale = quant\_scale.view(model.weight.dtype)  
> 270 quant\_min = quant\_min.view(model.weight.dtype)  
> 272 replaced\_old\_value = state\_dict[key]
> 
> RuntimeError: self.size(-1) must be divisible by 2 to view BFloat16 as Float (different element sizes), but got 1"  
> }

I tried setting the group\_dim to 2 in the config.json, but this gave the error that the tuple was out of range.

My GPU doesn’t support BFloat type format. It is by default disabled I thought, but specifically set the dtype to fp32 (not that this changes anything). How can I fix this issue? The reason why I would like to use zero stage 3 is to go from a 2B model to a 7B model, offloading to CPU (also to test this out a bit).