# GPT2 Implementation from scratch

**URL:** <https://discuss.huggingface.co/t/gpt2-implementation-from-scratch/679>\
**Category:** 🤗Transformers\
**Created:** [August 11, 2020, 3:00pm UTC](https://discuss.huggingface.co/t/gpt2-implementation-from-scratch/679 "2020-08-11T15:00:36Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jisng-prk](https://avatars.discourse-cdn.com/v4/letter/j/d78d45/32.png) [@jisng-prk](https://discuss.huggingface.co/u/jisng-prk)\
**Post date:** [August 11, 2020, 3:00pm UTC](https://discuss.huggingface.co/t/gpt2-implementation-from-scratch/679/1 "2020-08-11T15:00:36Z")

</div>

Sorry for ask about same question. But I need help.  
I am trying to understand GPT2 structure from bottom to top.

What I am having trouble is that why the causal masking has such variable like ns, nd.  
I think the nd and ns is always same because they were splited from same ‘x’ input  
So, the nd and ns would be sequence length. Is it right?

The lower text is more detailed question. Thank you for help

* * *

In transformers/src/transformers/modeling\_gpt2.py .

what is nd, ns variable in line 150

```auto
def _attn(self, q, k, v, attention_mask=None, head_mask=None, output_attentions=False):
    w = torch.matmul(q, k)
    if self.scale:
        w = w / (float(v.size(-1)) ** 0.5)
    nd, ns = w.size(-2), w.size(-1)
    mask = self.bias[:, :, ns - nd : ns, :ns]

```

because gpt model performs self-attention, isn’t the nd and ns always same?  
What is the meaning of “ns - nd: ns”

As you can see in line 187,

query, key, value = x.split(self.split\_size, dim=2)

the query and key should have same sequence length which is nd and ns

Thank you