# Parameter Count & Shape Discrepancies in 4-bit vs. Higher bit LLM models

**URL:** <https://discuss.huggingface.co/t/parameter-count-shape-discrepancies-in-4-bit-vs-higher-bit-llm-models/69512>\
**Category:** 🤗Transformers\
**Created:** [January 17, 2024, 4:26pm UTC](https://discuss.huggingface.co/t/parameter-count-shape-discrepancies-in-4-bit-vs-higher-bit-llm-models/69512 "2024-01-17T16:26:54Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![kdcyberdude](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kdcyberdude/32/27478_2.png) [@kdcyberdude](https://discuss.huggingface.co/u/kdcyberdude)\
**Post date:** [January 17, 2024, 4:26pm UTC](https://discuss.huggingface.co/t/parameter-count-shape-discrepancies-in-4-bit-vs-higher-bit-llm-models/69512/1 "2024-01-17T16:26:54Z")

</div>

I’m delving into an intriguing issue related to model quantization and seeking insights from the community. Specifically, I’ve observed a curious difference in the parameter count and shape between 4-bit and 8/16/32-bit models.

I loaded the same model in different bit representations. Intriguingly, the 4-bit model shows different parameter counts and shapes compared to its 8/16/32-bit counterparts. Checked on Phi and Llama.

```auto
m1 = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", trust_remote_code=True, load_in_4bit=True)
m2 = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype="auto", trust_remote_code=True, load_in_8bit=True)

print(list(m1.parameters())[1].shape, list(m2.parameters())[1].shape)
# Output: (torch.Size([3276800, 1]), torch.Size([2560, 2560]))

```

Why does the 4-bit model exhibit different parameter shapes and counts in comparison to higher-bit models? Is this a typical result of the quantization process or something else?

I found a related discussion that hints at quantization impacting trainable parameters [Less Trainable Parameters after quantization](https://discuss.huggingface.co/t/less-trainable-parameters-after-quantization/43192)

Any insights or theories on why this discrepancy occurs would be incredibly valuable.

Looking forward to your thoughts and discussions!

---

<div class="post-metadata">

**Author:** ![winnieyangwannan](https://avatars.discourse-cdn.com/v4/letter/w/8e7dd6/32.png) [@winnieyangwannan](https://discuss.huggingface.co/u/winnieyangwannan)\
**Post date:** [June 3, 2024, 6:19am UTC](https://discuss.huggingface.co/t/parameter-count-shape-discrepancies-in-4-bit-vs-higher-bit-llm-models/69512/2 "2024-06-03T06:19:12Z")

</div>

I have the same question. Have you get an answer yet? @kdcyberdude

---

<div class="post-metadata">

**Author:** ![kdcyberdude](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kdcyberdude/32/27478_2.png) [@kdcyberdude](https://discuss.huggingface.co/u/kdcyberdude)\
**Post date:** [June 3, 2024, 12:09pm UTC](https://discuss.huggingface.co/t/parameter-count-shape-discrepancies-in-4-bit-vs-higher-bit-llm-models/69512/3 "2024-06-03T12:09:26Z")

</div>

Hi @winnieyangwannan,

To explain briefly, it’s due to the packaging of 4-bit quantized weights. Essentially, one byte holds two parameters, each consisting of 4 bits.

For a more detailed explanation, feel free to check out this discussion: [Hugging Face Forum - Impact of Quantization on Trainable Parameters](https://discuss.huggingface.co/t/less-trainable-parameters-after-quantization/43192/12).