# Why BertForMaskedLM has decoder layer

**URL:** <https://discuss.huggingface.co/t/why-bertformaskedlm-has-decoder-layer/9263>\
**Category:** 🤗Transformers\
**Created:** [August 16, 2021, 9:13am UTC](https://discuss.huggingface.co/t/why-bertformaskedlm-has-decoder-layer/9263 "2021-08-16T09:13:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ccfeidao](https://avatars.discourse-cdn.com/v4/letter/c/f08c70/32.png) [@ccfeidao](https://discuss.huggingface.co/u/ccfeidao)\
**Post date:** [August 16, 2021, 9:13am UTC](https://discuss.huggingface.co/t/why-bertformaskedlm-has-decoder-layer/9263/1 "2021-08-16T09:13:14Z")

</div>

I want to start a new pre training language. When I use ’bertformaskedLM‘, I find that the model has a decoder layer, and the output of the model should be 768 dimensions, not 20000 (20000 is the number of tokens). What is the reason for this problem ？and how can I pretrain a language correctly？ thank you.

this code:

```auto
config = BertConfig(
    vocab_size=20000,
    hidden_size=768, 
    num_hidden_layers=6, 
    num_attention_heads=12,
    max_position_embeddings=512
)

model = BertForMaskedLM(config)

training_args = TrainingArguments(
    output_dir='./bangla_data/working/',
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_device_train_batch_size=32,
    save_steps=1000, #10000
    save_total_limit=2,
    prediction_loss_only=True,
    learning_rate: float = 5e-05,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)

```

model = BertForMaskedLM(config)  
print('No of parameters: ', model.num\_parameters())

this model:  
BertForMaskedLM(  
(bert): BertModel(  
(embeddings): BertEmbeddings(  
(word\_embeddings): Embedding(20000, 768, padding\_idx=0)  
(position\_embeddings): Embedding(512, 768)  
(token\_type\_embeddings): Embedding(2, 768)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
(encoder): BertEncoder(  
(layer): ModuleList(  
(0): BertLayer(  
(attention): BertAttention(  
(self): BertSelfAttention(  
(query): Linear(in\_features=768, out\_features=768, bias=True)  
(key): Linear(in\_features=768, out\_features=768, bias=True)  
(value): Linear(in\_features=768, out\_features=768, bias=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
(output): BertSelfOutput(  
(dense): Linear(in\_features=768, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(intermediate): BertIntermediate(  
(dense): Linear(in\_features=768, out\_features=3072, bias=True)  
)  
(output): BertOutput(  
(dense): Linear(in\_features=3072, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(1): BertLayer(  
(attention): BertAttention(  
(self): BertSelfAttention(  
(query): Linear(in\_features=768, out\_features=768, bias=True)  
(key): Linear(in\_features=768, out\_features=768, bias=True)  
(value): Linear(in\_features=768, out\_features=768, bias=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
(output): BertSelfOutput(  
(dense): Linear(in\_features=768, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(intermediate): BertIntermediate(  
(dense): Linear(in\_features=768, out\_features=3072, bias=True)  
)  
(output): BertOutput(  
(dense): Linear(in\_features=3072, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(2): BertLayer(  
(attention): BertAttention(  
(self): BertSelfAttention(  
(query): Linear(in\_features=768, out\_features=768, bias=True)  
(key): Linear(in\_features=768, out\_features=768, bias=True)  
(value): Linear(in\_features=768, out\_features=768, bias=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
(output): BertSelfOutput(  
(dense): Linear(in\_features=768, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(intermediate): BertIntermediate(  
(dense): Linear(in\_features=768, out\_features=3072, bias=True)  
)  
(output): BertOutput(  
(dense): Linear(in\_features=3072, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(3): BertLayer(  
(attention): BertAttention(  
(self): BertSelfAttention(  
(query): Linear(in\_features=768, out\_features=768, bias=True)  
(key): Linear(in\_features=768, out\_features=768, bias=True)  
(value): Linear(in\_features=768, out\_features=768, bias=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
(output): BertSelfOutput(  
(dense): Linear(in\_features=768, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(intermediate): BertIntermediate(  
(dense): Linear(in\_features=768, out\_features=3072, bias=True)  
)  
(output): BertOutput(  
(dense): Linear(in\_features=3072, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(4): BertLayer(  
(attention): BertAttention(  
(self): BertSelfAttention(  
(query): Linear(in\_features=768, out\_features=768, bias=True)  
(key): Linear(in\_features=768, out\_features=768, bias=True)  
(value): Linear(in\_features=768, out\_features=768, bias=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
(output): BertSelfOutput(  
(dense): Linear(in\_features=768, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(intermediate): BertIntermediate(  
(dense): Linear(in\_features=768, out\_features=3072, bias=True)  
)  
(output): BertOutput(  
(dense): Linear(in\_features=3072, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(5): BertLayer(  
(attention): BertAttention(  
(self): BertSelfAttention(  
(query): Linear(in\_features=768, out\_features=768, bias=True)  
(key): Linear(in\_features=768, out\_features=768, bias=True)  
(value): Linear(in\_features=768, out\_features=768, bias=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
(output): BertSelfOutput(  
(dense): Linear(in\_features=768, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
(intermediate): BertIntermediate(  
(dense): Linear(in\_features=768, out\_features=3072, bias=True)  
)  
(output): BertOutput(  
(dense): Linear(in\_features=3072, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
(dropout): Dropout(p=0.1, inplace=False)  
)  
)  
)  
)  
)  
(cls): BertOnlyMLMHead(  
(predictions): BertLMPredictionHead(  
(transform): BertPredictionHeadTransform(  
(dense): Linear(in\_features=768, out\_features=768, bias=True)  
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise\_affine=True)  
)  
(decoder): Linear(in\_features=768, out\_features=20000, bias=True)  
)  
)  
)

---

<div class="post-metadata">

**Author:** ![lvwerra](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lvwerra/32/2427_2.png) [@lvwerra](https://discuss.huggingface.co/u/lvwerra)\
**Post date:** [August 17, 2021, 9:09am UTC](https://discuss.huggingface.co/t/why-bertformaskedlm-has-decoder-layer/9263/2 "2021-08-17T09:09:27Z")

</div>

Hi @ccfeidao

For your first question about the decoder and the hidden size and output size: Internally, the model projects the input tokens which have dimension of the vocabulary size (20’000 in your case) to the hidden size (768 in your case). Inside the layers of BERT the embeddings of 768 are processed. Finally, after the last BERT layer we need to get back from the hidden size to the vocabulary size which corresponds to proper tokens. That’s what the decoder layer does: it takes embeddings of `dim=768` and projects them to `dim=20000`.

As for your second question about pretraining: there is a tutorial on [Google Colab](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb).

---

<div class="post-metadata">

**Author:** ![ccfeidao](https://avatars.discourse-cdn.com/v4/letter/c/f08c70/32.png) [@ccfeidao](https://discuss.huggingface.co/u/ccfeidao)\
**Post date:** [August 17, 2021, 9:38am UTC](https://discuss.huggingface.co/t/why-bertformaskedlm-has-decoder-layer/9263/3 "2021-08-17T09:38:50Z")

</div>

Thank you very much.I get it