# What is ns, nd variable

**URL:** <https://discuss.huggingface.co/t/what-is-ns-nd-variable/647>\
**Category:** Beginners\
**Created:** [August 9, 2020, 8:16am UTC](https://discuss.huggingface.co/t/what-is-ns-nd-variable/647 "2020-08-09T08:16:03Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jisng-prk](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@jisng-prk](https://discuss.huggingface.co/u/jisng-prk)\
**Post date:** [August 9, 2020, 8:16am UTC](https://discuss.huggingface.co/t/what-is-ns-nd-variable/647/1 "2020-08-09T08:16:03Z")

</div>

In transformers/src/transformers/modeling\_gpt2.py .

what is nd, ns variable in line 150

```
def _attn(self, q, k, v, attention_mask=None, head_mask=None, output_attentions=False):
    w = torch.matmul(q, k)
    if self.scale:
        w = w / (float(v.size(-1)) ** 0.5)
    nd, ns = w.size(-2), w.size(-1)
    mask = self.bias[:, :, ns - nd : ns, :ns]

```

because gpt model performs self-attention, isn’t the nd and ns always same?  
What is the meaning of “ns - nd: ns”

As you can see in line 187,

query, key, value = x.split(self.split\_size, dim=2)

the query and key should have same sequence length which is nd and ns

Thank you