# Question about all\_head\_size under BertSelfAttention

**URL:** <https://discuss.huggingface.co/t/question-about-all-head-size-under-bertselfattention/263>\
**Category:** 🤗Transformers\
**Created:** [July 13, 2020, 8:49pm UTC](https://discuss.huggingface.co/t/question-about-all-head-size-under-bertselfattention/263 "2020-07-13T20:49:51Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ndharap](https://avatars.discourse-cdn.com/v4/letter/n/41988e/32.png) [@ndharap](https://discuss.huggingface.co/u/ndharap)\
**Post date:** [July 13, 2020, 8:49pm UTC](https://discuss.huggingface.co/t/question-about-all-head-size-under-bertselfattention/263/1 "2020-07-13T20:49:51Z")

</div>

While going through the codebase , I found the following code under the BertSelfAttention class here - ([BertSelfAttention](https://huggingface.co/transformers/_modules/transformers/modeling_bert.html#BertModel)-

```
self.num_attention_heads = config.num_attention_heads
self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
self.all_head_size = self.num_attention_heads * self.attention_head_size

```

I am not sure I understand the reason for calculating all\_head\_size again instead of assigning it the value of config.hidden\_size directly.

Am I missing something?

I also see a check right above this that ensures that the hidden\_size is divisible by num\_attention\_heads:

```
    if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"):
        raise ValueError(
            "The hidden size (%d) is not a multiple of the number of attention "
            "heads (%d)" % (config.hidden_size, config.num_attention_heads)
        )
```