# Difference between transformer encoder and decoder **URL:** \ **Category:** Models\ **Created:** [March 3, 2021, 6:20pm UTC](https://discuss.huggingface.co/t/difference-between-transformer-encoder-and-decoder/4127 "2021-03-03T18:20:26Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![voidful](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/voidful/32/27454_2.png) [@voidful](https://discuss.huggingface.co/u/voidful)\ **Post date:** [March 3, 2021, 6:20pm UTC](https://discuss.huggingface.co/t/difference-between-transformer-encoder-and-decoder/4127/1 "2021-03-03T18:20:26Z")
I am trying to understand the difference between transformer encoder and decoder, after reading the article [Transformer-based Encoder-Decoder Models](https://huggingface.co/blog/encoder-decoder#decoder) . **Would it be correct that after bringing a causal masked to encoder only model, it will be the same as decoder only model？** according to the article: > _auto-regressive_ models, such as GPT2, have the same architecture as _transformer-based_ decoder models **if** one removes the cross-attention layer > On a side-note, _autoencoding_ models, such as Bert, have the same architecture as _transformer-based_ encoder models. So, without involving cross-attention, the main difference between transformer encoder and decoder is that encoder uses bi-directional self-attention, decoder uses uni-directional self-attention layer instead. BERT is an encoder-only model and GPT is a decoder-only model. What if I add a causal mask on BERT model to make it become decoder. Refer to the extended attention mask on Bert. It can be done by changing the config with is\_decoder. > > After that, I have an experiment comparing the values of word embedding of “I” for input\_ids and perturbed input\_ids as same as the article. **It seems that after changing BERT to a decoder, its hidden state will be changed on a different input, why is that happen ?** ```python from transformers import AutoModel,AutoTokenizer, AutoConfig import torch # bert model_config = AutoConfig.from_pretrained('bert-base-uncased') model_config.is_decoder = True bert_model = AutoModel.from_config(model_config) bert_tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased') # gpt gpt_model = AutoModel.from_pretrained('gpt2') gpt_tokenizer = AutoTokenizer.from_pretrained('gpt2') ``` Test GPT model ```python embeddings = gpt_model.get_input_embeddings() # create ids of encoded input vectors decoder_input_ids = gpt_tokenizer(" Ich will ein", return_tensors="pt", add_special_tokens=False).input_ids # pass decoder input_ids and encoded input vectors to decoder lm_logits = gpt_model(decoder_input_ids).last_hidden_state # change the decoder input slightly decoder_input_ids_perturbed = gpt_tokenizer(" Ich will das", return_tensors="pt", add_special_tokens=False).input_ids lm_logits_perturbed = gpt_model(decoder_input_ids_perturbed).last_hidden_state # compare values of word embedding of "I" for input_ids and perturbed input_ids print("Is encoding for `Ich` equal to its perturbed version?: ", torch.allclose(lm_logits[0, 0], lm_logits_perturbed[0, 0], atol=1e-3)) ``` `Is encoding for `Ich` equal to its perturbed version?: True` Test BERT model ```python embeddings = bert_model.get_input_embeddings() # create ids of encoded input vectors decoder_input_ids = bert_tokenizer(" Ich will ein", return_tensors="pt", add_special_tokens=False).input_ids # pass decoder input_ids and encoded input vectors to decoder lm_logits = bert_model(decoder_input_ids).last_hidden_state # change the decoder input slightly decoder_input_ids_perturbed = bert_tokenizer(" Ich will das", return_tensors="pt", add_special_tokens=False).input_ids lm_logits_perturbed = bert_model(decoder_input_ids_perturbed).last_hidden_state # compare values of word embedding of "I" for input_ids and perturbed input_ids print("Is encoding for `Ich` equal to its perturbed version?: ", torch.allclose(lm_logits[0, 0], lm_logits_perturbed[0, 0], atol=1e-3)) ``` `Is encoding for `Ich` equal to its perturbed version?: False` ---
**Author:** ![voidful](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/voidful/32/27454_2.png) [@voidful](https://discuss.huggingface.co/u/voidful)\ **Post date:** [March 12, 2021, 3:26pm UTC](https://discuss.huggingface.co/t/difference-between-transformer-encoder-and-decoder/4127/2 "2021-03-12T15:26:22Z")
It is because of the dropout. > > > \# Problem > Causal Models is only attended to the left context. Therefore causal… models should not depend on the right tokens. > For example, The word embedding of "I" will be unchanged no matter what is in the right In GPT2. Since Causal Language Model are uni-directional self-attention. > > \`\`\` > from transformers import AutoModel,AutoTokenizer, AutoConfig > import torch > > \# gpt > gpt\_model = AutoModel.from\_pretrained('gpt2') > gpt\_tokenizer = AutoTokenizer.from\_pretrained('gpt2') > embeddings = gpt\_model.get\_input\_embeddings() > > \# create ids of encoded input vectors > decoder\_input\_ids = gpt\_tokenizer("\ Ich will ein", return\_tensors="pt", add\_special\_tokens=False).input\_ids > > \# pass decoder input\_ids and encoded input vectors to decoder > lm\_logits = gpt\_model(decoder\_input\_ids).last\_hidden\_state > > \# change the decoder input slightly > decoder\_input\_ids\_perturbed = gpt\_tokenizer("\ Ich will das", return\_tensors="pt", add\_special\_tokens=False).input\_ids > lm\_logits\_perturbed = gpt\_model(decoder\_input\_ids\_perturbed).last\_hidden\_state > > \# compare values of word embedding of "I" for input\_ids and perturbed input\_ids > print("Is encoding for \`Ich\` equal to its perturbed version?: ", torch.allclose(lm\_logits\[0, 0\], lm\_logits\_perturbed\[0, 0\], atol=1e-3)) > \`\`\` > Result > \`\`\` > Is encoding for \`Ich\` equal to its perturbed version?: True > \`\`\` > > However, when it comes to other models, the result is not following the assumption, the logits will be changed when changing the right side input? What is the reason? Is it a bug? I really want to know the answer, thank you! > > BERT > \`\`\` > Is encoding for \`Ich\` equal to its perturbed version?: False > \`\`\` > > BART > \`\`\` > Is encoding for \`Ich\` equal to its perturbed version?: False > \`\`\` > > Roberta > \`\`\` > Is encoding for \`Ich\` equal to its perturbed version?: False > \`\`\` > > Experiment notebook \[colab\](https://colab.research.google.com/drive/15V37RWAL40vhrk-uBIh9m99j1gZMLjUy?usp=sharing) > > \## Environment info > > \- \`transformers\` version: 4.3.3 > \- Platform: Linux-4.19.112+-x86\_64-with-Ubuntu-18.04-bionic > \- Python version: 3.7.10 > \- PyTorch version (GPU?): 1.7.1+cu101 (False) > \- Tensorflow version (GPU?): 2.4.1 (False) > \- Using GPU in script?: \ > \- Using distributed or parallel set-up in script?: \ > > \### Who can help > > \- @patrickvonplaten > \- @LysandreJik > \- @patil-suraj > > > \